Nearest Linear Manifold Classification

Yh Yao,Lh Chen,Yq Chen

2001-01-01

Abstract:A novel classifier, named Nearest Linear Manifold uses a small number of prototypes to represent a class and extend their resentational capacity by using the linear manifold of the prototypes to provide more sufficient feature information for classification.

What problem does this paper attempt to address?

Learning Exemplar-Represented Manifolds in Latent Space for Classification.

Shu Kong,Donghui Wang

DOI: https://doi.org/10.1007/978-3-642-40994-3_16

2013-01-01

Abstract:Intrinsic manifold structure of a data collection is valuable information for classification task. By considering the manifold structure in the data set for classification and with the sparse coding framework, we propose an algorithm to: (1) find exemplars from each class to represent the class-specific manifold structure, in which way the object-space dimensionality is reduced; (2) simultaneously learn a latent feature space to make the mapped data more discriminative according to the class-specific manifold measurement. We call the proposed algorithm Exemplar-represented Manifold in Latent Space for Classification (EMLSC). We also present the nonlinear extension of EMLSC based on kernel tricks to deal with highly nonlinear situations. Experiments on synthetic and real-world datasets demonstrate the merit of the proposed method.
A Novel Image Classification Method Based on Manifold Learning and Gaussian Mixture Model

Xianjun Zhang,Min Yao,Rong Zhu

DOI: https://doi.org/10.1109/iasp.2010.5476120

2010-01-01

Abstract:Image classification is one of the important parts of digital image processing. We propose a novel feature space-based image classification method by combining manifold learning and mixture model. In this paper, the process of image classification can be viewed as two parts: a coarse-grained classification and a fine-grained classification. In the coarse-grained classification, we apply the ISOMAP (Isometric Mapping) algorithm to do a dimensional reduction based on manifold learning. Thus, solving the classification problem is transformed from a high-dimensional data space to a low-dimensional feature space. And then, during the fine-grained classification, we present an improved EM algorithm of finite Gaussian mixture model to do clustering. Experimental results have demonstrated that the proposed method performs well in both accuracy and time. Additionally, our algorithm is robust to some extent.
Classification Using Distances from Samples to Linear Manifolds.

Yiguang Liu,Xiaochun Cao,Jian Guo Liu

DOI: https://doi.org/10.1007/s10044-011-0242-x

IF: 2.307

2011-01-01

Pattern Analysis and Applications

Abstract:A classifier is proposed wherein the distances from samples to linear manifolds (DSL) are used to perform classification. For each class, a linear manifold is built, whose dimension is high enough to pass all the training samples of the class. The distance from a query sample to a linear manifold is converted to the distance from a point to a linear subspace. And a simple and stable formula is derived to calculate the distance by virtue of the geometrical fundamental of the Gram matrix as well as the regularization technique. The query sample is assigned into the class whose linear manifold is the nearest. On one synthetic data set, thirteen binary-class data sets as well as six multi-class data sets, the experimental results show that the classification performance of DSL is of competence. On most of the data sets, DSL outperforms the comparing classifiers based on k nearest samples or subspaces, and is even superior to support vector machines on some data sets. Further experiment demonstrates that the test efficiency of DSL is also competitive to k NN and the related state-of-the-art classifiers on many data sets.
K-Nearest Neighbors Classifier over Manifolds

WEN Zhi-qiang,HU Yong-xiang,ZHU Wen-qiu

DOI: https://doi.org/10.3724/sp.j.1087.2012.03311

2012-01-01

Abstract:For resolving the problem of the existing noise sample and large number of dimensions,the k-nearest neighbors classifier over manifolds was presented in this paper.Firstly the classic k-nearest neighbors was extended by Bayes theorem and local joint probability density was estimated by kernel density estimation in classifier.In addition,after building the noise sample model,an objective function was defined via improved marginal intrinsic graph and its weight matrix for searching the optimal dimension reduction mapping matrix.At last,details about k-nearest neighbors algorithm over manifolds were provided.The experimental results demonstrate that the presented method has lower classification error rate than six kinds of classic methods in most cases on twelve data sets.
Nearest Manifold Approach for Face Recognition

JP Zhang,SZ Li,J Wang

DOI: https://doi.org/10.1109/afgr.2004.1301535

2004-01-01

Abstract:Faces under varying illumination, pose and non-rigid deformation are empirically thought of as a highly nonlinear manifold in the observation space. How to discover intrinsic low-dimensional manifold is important to characterize meaningful face distributions and classify them using a simpler, such as linear or Gaussian based, classifier. In this paper, we present a manifold learning algorithm (MLA) for learning a mapping from highly-dimensional manifold into the intrinsic low-dimensional linear manifold. We also propose the nearest manifold (NM) criterion for the classification and present an algorithm for computing the distance from the sample to be classified to the nearest face manifolds in light of local linearity of manifold. Based on these works, face recognition is achieved with the combination of MLA and NM. Experiments on several face databases show that the advantages of our proposed combinational approach.
Local Manifold Matching for Face Recognition

W Liu,W Fan,YH Wang,TN Tan

DOI: https://doi.org/10.1109/icip.2005.1530208

2005-01-01

Abstract:In this paper, we propose a novel classification method, called local manifold matching (LMM), for face recognition. LMM has great representational capacity of available prototypes and is based on the local linearity assumption that each data point and its k nearest neighbors from the same class lie on a linear manifold locally embedded in the image space. We present a supervised local manifold learning algorithm for learning all locally linear manifold structures. Then we propose the nearest manifold criterion for the classification in which the query feature point is assigned to the most matching face manifold. Experimental results show that kernel PCA incorporated with the LMM classifier achieves the best face recognition performance.
Learning nonlinear manifolds based on mixtures of localized linear manifolds under a self-organizing framework

Huicheng Zheng,Wei Shen,Qionghai Dai,Sanqing Hu,Zhe-Ming Lu

DOI: https://doi.org/10.1016/j.neucom.2009.01.008

IF: 6

2009-01-01

Neurocomputing

Abstract:This paper presents a neural model which learns low-dimensional nonlinear manifolds embedded in higher-dimensional data space based on mixtures of local linear manifolds under a self-organizing framework. Compared to other similar networks, the local linear manifolds learned by our network have a more localized representation of local data distributions thanks to a new distortion measure, which removes confusion between sub-models that exists in many similar mixture models. Each neuron in the network asymptotically learns a mean vector and a principal subspace of the data in its local region. It is proved that there is no local extremum for each sub-model. Experiments show that the new mixture model is better adapted to nonlinear manifolds of various data distributions than other similar models. The online-learning property of this model is desirable when the data set is very large, when computational efficiency is of paramount importance, or when data are sequentially input. We further show an application of this model to recognition of handwritten digit images based on mixtures of local linear manifolds.
Soft-margin classification of object manifolds

Uri Cohen,Haim Sompolinsky

DOI: https://doi.org/10.1103/PhysRevE.106.024126

2022-03-14

Abstract:A neural population responding to multiple appearances of a single object defines a manifold in the neural response space. The ability to classify such manifolds is of interest, as object recognition and other computational tasks require a response that is insensitive to variability within a manifold. Linear classification of object manifolds was previously studied for max-margin classifiers. Soft-margin classifiers are a larger class of algorithms and provide an additional regularization parameter used in applications to optimize performance outside the training set by balancing between making fewer training errors and learning more robust classifiers. Here we develop a mean-field theory describing the behavior of soft-margin classifiers applied to object manifolds. Analyzing manifolds with increasing complexity, from points through spheres to general manifolds, a mean-field theory describes the expected value of the linear classifier's norm, as well as the distribution of fields and slack variables. By analyzing the robustness of the learned classification to noise, we can predict the probability of classification errors and their dependence on regularization, demonstrating a finite optimal choice. The theory describes a previously unknown phase transition, corresponding to the disappearance of a non-trivial solution, thus providing a soft version of the well-known classification capacity of max-margin classifiers.

Disordered Systems and Neural Networks,Statistical Mechanics,Neural and Evolutionary Computing,Neurons and Cognition,Machine Learning
Joint Nuclear-Norm Nonlinear Manifold Learning And Robust Classification By Linear Embedding

Yan Zhang,Zhao Zhang,Lei Jia,Fanzhang Li

DOI: https://doi.org/10.1109/IJCNN.2016.7727684

2016-01-01

Abstract:We propose a Joint nuclear norm based nonlinear Manifold Learning through linear embedding with Classification, called JMLC. By including a feature approximation error into the existing nonlinear manifold learning framework to correlate manifold features with embedded features by a linear projection, the learnt projection can handle the outside points efficiently by embedding. Besides, to encode the neighborhood reconstruction error in manifold learning part, we apply a more reliable nuclear norm based distance metric, since nuclear norm is proved to be more reliable than both L-1-norm and Frobenius norm. To make learnt nonlinear features be optimal for classification so that the accuracy can be enhanced, we also minimize a robust L-2,L-1-norm based regressive classification error over the embedded manifold features, which ensures the classification process to be robust to noise and outliers in data. Based on performing joint manifold learning and classification alternately, our JMLC can obtain a low-dimensional embedding, a linear projection and a multi-class classifier simultaneously. Extensive results demonstrate the validity of our proposed algorithm for feature extraction and robust classification, compared with other related models.
Nearest Feature Line Classifier Based on Collaborative Representation with Nearest Neighbour Search Algorithm

Zhongjie Zhuang,Jeng-Shyang Pan,Shu-Chuan Chu,Hao Luo

DOI: https://doi.org/10.1049/ell2.12030

2020-01-01

Electronics Letters

Abstract:Nearest feature line is an effective classification algorithm. However, if a test sample cannot be linear represented by the training samples, the algorithm may not work very well. Moreover, another problem is that it will have a large computation complexity. Therefore, the authors propose a novel algorithm. To begin with, the test sample is linear represented by all the training samples, and the errors between the test sample and every training sample are calculated. The authors only keep the training samples with small errors. In this way, on the one hand, training samples are not suitable for the test sample will be ignored, on the other hand, running time can be reduced. To further reduce the computing time of the algorithm, nearest neighbour search technique is applied to the algorithm. Experiments on numerical and image database show the algorithm cannot only improve the classification accuracy, but also reduce runtime.
A discriminative multiple-manifold network for image set classification

Hao Wu,Weigang Wang,Zishan Xia,Yonghao Chen,Yuanjian Liu,Jianfei Chen,Wu, Hao,Wang, Weigang,Xia, Zishan,Chen, Yonghao

DOI: https://doi.org/10.1007/s10489-023-04900-1

IF: 5.3

2023-08-04

Applied Intelligence

Abstract:Because the distinct advantages of manifold-learning methods for feature extraction, Riemannian manifolds have been used extensively in image recognition tasks in recent years. However, large intra-class variation is a major problem for researchers. Although recent studies have alleviated this problem by extending nonlinear learning to Riemannian manifolds, important classification information may be lost when describing image sets with a single manifold. To address this problem, we design a novel discriminative multiple-manifold network (DMMNet) for image recognition. The Grassmann and symmetric positive definite manifold are first used simultaneously to model each image set. Then, the symmetric positive definite manifold and Grassmann manifold nonlinear learning blocks are devised to transform the data into low-dimensional manifold matrices to enhance their discriminability. A feature fusion layer is designed to fuse the complementary features of different manifolds, and a point-to-point hidden layer is devised to focus on more important feature maps. To acquire the importance of the manifold feature maps, a feature selection method based on manifold feature importance is designed to learn the corresponding weight coefficients. Finally, an output layer is used to map the optimal feature subset into the Euclidean space, such that the kernel discriminant analysis and K-nearest neighbor algorithm are applied. The proposed DMMNet is compared with state-of-the-art techniques on four datasets. Extensive experimental results demonstrate the effectiveness of the proposed DMMNet.

computer science, artificial intelligence
A Graph-Based Semi-Supervised K Nearest-Neighbor Method for Nonlinear Manifold Distributed Data Classification.

Enmei Tu,Yaqian Zhang,Lin Zhu,Jie Yang,Nikola Kasabov

DOI: https://doi.org/10.1016/j.ins.2016.07.016

IF: 8.1

2016-01-01

Information Sciences

Abstract:k Nearest Neighbors (kNN) is one of the most widely used supervised learning algorithms to classify Gaussian distributed data, but it does not achieve good results when it is applied to nonlinear manifold distributed data, especially when a very limited amount of labeled samples are available. In this paper, we propose a new graph-based kNN algorithm which can effectively handle both Gaussian distributed data and nonlinear manifold distributed data. To achieve this goal, we first propose a constrained Tired Random Walk (TRW) by constructing an R-level nearest-neighbor strengthened tree over the graph, and then compute a TRW matrix for similarity measurement purposes. After this, the nearest neighbors are identified according to the TRW matrix and the class label of a query point is determined by the sum of all the TRW weights of its nearest neighbors. To deal with online situations, we also propose a new algorithm to handle sequential samples based a local neighborhood reconstruction. Comparison experiments are conducted on both synthetic data sets and real-world data sets to demonstrate the validity of the proposed new kNN algorithm and its improvements to other version of kNN algorithms. Given the widespread appearance of manifold structures in real-world problems and the popularity of the traditional kNN algorithm, the proposed manifold version kNN shows promising potential for classifying manifold-distributed data.
Distance Learner: Incorporating Manifold Prior to Model Training

Aditya Chetan,Nipun Kwatra

DOI: https://doi.org/10.48550/arXiv.2207.06888

2022-07-14

Abstract:The manifold hypothesis (real world data concentrates near low-dimensional manifolds) is suggested as the principle behind the effectiveness of machine learning algorithms in very high dimensional problems that are common in domains such as vision and speech. Multiple methods have been proposed to explicitly incorporate the manifold hypothesis as a prior in modern Deep Neural Networks (DNNs), with varying success. In this paper, we propose a new method, Distance Learner, to incorporate this prior for DNN-based classifiers. Distance Learner is trained to predict the distance of a point from the underlying manifold of each class, rather than the class label. For classification, Distance Learner then chooses the class corresponding to the closest predicted class manifold. Distance Learner can also identify points as being out of distribution (belonging to neither class), if the distance to the closest manifold is higher than a threshold. We evaluate our method on multiple synthetic datasets and show that Distance Learner learns much more meaningful classification boundaries compared to a standard classifier. We also evaluate our method on the task of adversarial robustness, and find that it not only outperforms standard classifier by a large margin, but also performs at par with classifiers trained via state-of-the-art adversarial training.

Machine Learning,Artificial Intelligence
Neighbourhood Sensitive Preserving Embedding for Pattern Classification.

Bing-Hui Wang,Chuang Lin,Xue-Feng Zhao,Zhe-Ming Lu

DOI: https://doi.org/10.1049/iet-ipr.2013.0539

IF: 2.3

2014-01-01

IET Image Processing

Abstract:Recently, a large family of supervised or unsupervised manifold learning algorithms that stem from statistical or geometrical theory has been designed to solve the problem of pattern classification. In this study, consider the fact that the data are usually sampled from a low-dimensional manifold space which resides in a high-dimensional Euclidean space, the authors propose a novel two-graph-based supervised linear classification algorithm called neighbourhood sensitive preserving embedding (NSPE). Different from local linear embedding (LLE) (or neighbourhood preserving embedding (NPE)) which preserves the local neighbourhood structure with one graph, NSPE can discover both the intrinsic and discriminant structure of the data manifold by constructing two graphs, that is, the within-class graph and the between-class graph. Thus, the data are mapped into a subspace where the nearby points with the same label are close to each other, whereas the nearby points with different labels are far apart. As a classification method, besides being defined on training samples, NSPE is also defined on testing samples. Experiments carried on the real-world face databases demonstrate that the results of all two-graph-based spectral methods are comparable and better than that of one-graph-based methods.
PointManifold: Using Manifold Learning for Point Cloud Classification

Dinghao Yang,Wei Gao

DOI: https://doi.org/10.48550/arXiv.2010.07215

2020-10-16

Abstract:In this paper, we propose a point cloud classification method based on graph neural network and manifold learning. Different from the conventional point cloud analysis methods, this paper uses manifold learning algorithms to embed point cloud features for better considering the geometric continuity on the surface. Then, the nature of point cloud can be acquired in low dimensional space, and after being concatenated with features in the original three-dimensional (3D)space, both the capability of feature representation and the classification network performance can be improved. We pro-pose two manifold learning modules, where one is based on locally linear embedding algorithm, and the other is a non-linear projection method based on neural network architecture. Both of them can obtain better performances than the state-of-the-art baseline. Afterwards, the graph model is constructed by using the k nearest neighbors algorithm, where the edge features are effectively aggregated for the implementation of point cloud classification. Experiments show that the proposed point cloud classification methods obtain the mean class accuracy (mA) of 90.2% and the overall accuracy (oA)of 93.2%, which reach competitive performances compared with the existing state-of-the-art related methods.

Computer Vision and Pattern Recognition
Grassmann manifold for nearest points image set classification

Hengliang Tan,Zhengming Ma,Sumin Zhang,Zengrong Zhan,Beibei Zhang,Chenggong Zhang

DOI: https://doi.org/10.1016/j.patrec.2015.09.008

IF: 4.757

2015-12-01

Pattern Recognition Letters

Abstract:Image set classification has attracted increasing attention in recent years. How to effectively represent image sets is one key issue of set based classification. Subspaces form non-Euclidean Riemannian manifolds known as Grassmann manifolds, which allows an image set to be conveniently represented as a point on a Grassmann manifold is widely used in many visual classification tasks. Another issue is how to measure the distance/similarity between sets. Modeling image sets as hulls, and then finding distance of nearest points between sets as the set-to-set distance is a popular solution recently. In this paper, we propose a novel approach by exploiting the Projection kernel that explicitly maps the subspaces from the Grassmann manifold to a Reproducing Kernel Hilbert Space (RKHS) where the Euclidean geometry applies. And then, by modeling the points on RKHS as affine hulls, the Euclidean distance between the nearest points of two hulls can be used for classification. In order to obtain enough points for building the Grassmann affine hulls, we also develop a subspaces constructing method extended by K-means. Experiments are conducted on six datasets. Our proposed method achieves the best classification results on two multi-view object categorization datasets and one extreme illumination variation face recognition dataset.

computer science, artificial intelligence
Perceptual Nearest Neighbors for Classification

Guihua Wen,Jun Wen,Lijun Jiang

DOI: https://doi.org/10.1109/bicta.2010.5645347

2010-01-01

Abstract:Finding nearest neighbors plays a fundamental role in many artificial intelligence tasks, such as manifold learning, data mining, and information retrieval, etc. Directly applying this idea to perform classification is simple and often results in good performance on complex data types. However existing classifiers apply a well designed measure to find nearest neighbors. They still can not be comparable with human being in many complex cases such as on noisy, sparse or high dimensional data. This paper proposes a quite different but much interesting approach that utilizes Lipschitz function to define a simple topological transformation for modeling Gestalt laws of psychology from data and then designs a new measure to evaluate the quality of the discovered Gestalts. Subsequently, the nearest neighbors are selected from higher quality Gestalts, from which a new classifier is proposed that has much better classification performance.
Riemannian manifold learning based k-nearest-neighbor for hyperspectral image classification

Yushi Chen,Zhouhan Lin,Xing Zhao

DOI: https://doi.org/10.1109/IGARSS.2013.6723195

2013-01-01

Abstract:The existence of nonlinear characteristics in hyperspectral data is considered as an influential factor curtailing the classification accuracy of canonical linear classifier like k-nearest neighbor (k-NN). To deal with the problem, we investigated approaches to combine manifold learning methods and the k-NN classifier to preserve nonlinear characteristics contained in hyperspectral imagery. Then we proposed a Riemannian manifold learning (RML) based k-NN classifier for hyperspectral image classification, which substitutes the Euclidean distances used in canonical kNN by geodesic distances yielded by RML. The experimental results on AVIRIS data show that in most cases, the RML-kNN Classifier accesses higher classification accuracies than canonical k-NN.
A Supervised Manifold Learning Method

Zuojin Li,Weiren Shi,Xin Shi,Zhi Zhong

DOI: https://doi.org/10.2298/csis0902205l

2009-01-01

Computer Science and Information Systems

Abstract:The Locally Linear Embedding (LLE) algorithm is an unsupervised nonlinear dimensionality-reduction method, which reports a low recognition rate in classification because it gives no consideration to the label information of sample distribution. In this paper, a classification method of supervised LLE (SLLE) based on Linear Discriminant Analysis (LDA) is proposed. First, samples are classified according to their label values, and low dimensional features of intraclass data are expressed through LLE manifold learning. Then, the base vectors in Fisher subspace of the low dimensional features are generated through LDA learning. This method increases inter-class variation, and decreases the intra-class variation when samples are projected to the Fisher subspace. Hence, the samples of different labels can be recognized, and the recognition rate and robustness of the LLE learning are improved. Experiments on handwritten digit recognition show that the proposed method is featuring high recognition rate.
Few-shot classification based on manifold metric learning

Qingzhen Shang,Jinfu Yang,Jiaqi Ma,Jiahui Zhang

DOI: https://doi.org/10.1117/1.jei.33.1.013026

IF: 0.829

2024-01-24

Journal of Electronic Imaging

Abstract:Few-shot classification aims to classify samples with a limited quantity of labeled training data, and it can be widely applied in practical scenarios such as wastewater treatment plants and healthcare. Compared with traditional methods, existing deep metric-based algorithms have excelled in few-shot classification tasks, but some issues need to be further investigated. While current standard convolutional networks can extract expressive depth features, they do not fully exploit the relationships among input sample attributes. Two problems are included here: (1) how to extract more expressive features and transform them into attributes, and (2) how to obtain the optimal combination of sample class attributes. This paper proposes a few-shot classification method based on manifold metric learning (MML) with feature space embedded in symmetric positive definite (SPD) manifolds to overcome the above limitations. First, significant features are extracted using the proposed joint dynamic convolution module. Second, the definition and properties of Riemannian popular strictly convex geodesics are used to minimize the proposed MML loss function and obtain the optimal attribute correlation matrix A. We theoretically prove that the MML is popularly strictly convex in the SPD and obtain the global optimal solution in the closed space. Extensive experimental results on popular datasets show that our proposed approach outperforms other state-of-the-art methods.

engineering, electrical & electronic,optics,imaging science & photographic technology

Nearest Linear Manifold Classification

Learning Exemplar-Represented Manifolds in Latent Space for Classification.

A Novel Image Classification Method Based on Manifold Learning and Gaussian Mixture Model

Classification Using Distances from Samples to Linear Manifolds.

K-Nearest Neighbors Classifier over Manifolds

Nearest Manifold Approach for Face Recognition

Local Manifold Matching for Face Recognition

Learning nonlinear manifolds based on mixtures of localized linear manifolds under a self-organizing framework

Soft-margin classification of object manifolds

Joint Nuclear-Norm Nonlinear Manifold Learning And Robust Classification By Linear Embedding

Nearest Feature Line Classifier Based on Collaborative Representation with Nearest Neighbour Search Algorithm

A discriminative multiple-manifold network for image set classification

A Graph-Based Semi-Supervised K Nearest-Neighbor Method for Nonlinear Manifold Distributed Data Classification.

Distance Learner: Incorporating Manifold Prior to Model Training

Neighbourhood Sensitive Preserving Embedding for Pattern Classification.

PointManifold: Using Manifold Learning for Point Cloud Classification

Grassmann manifold for nearest points image set classification

Perceptual Nearest Neighbors for Classification

Riemannian manifold learning based k-nearest-neighbor for hyperspectral image classification

A Supervised Manifold Learning Method

Few-shot classification based on manifold metric learning