Learning latent features by nonnegative matrix factorization combining similarity judgments

ZhangJiang-She,WangChang-Peng,YangYu-Qian

IF: 6

2015-01-01

Neurocomputing

Abstract:Nonnegative matrix factorization (NMF) is a popular method for learning low-rank approximation of nonnegative matrix. However, aiming at seeking the low-rank approximation from the viewpoint of dat...

What problem does this paper attempt to address?

Feature Extraction Via Multi-View Non-Negative Matrix Factorization with Local Graph Regularization.

Zhenfan Wang,Xiangwei Kong,Haiyan Fu,Ming Li,Yujia Zhang

DOI: https://doi.org/10.1109/icip.2015.7351455

2015-01-01

Abstract:Feature extraction is a crucial and difficult issue in pattern recognition tasks with the high-dimensional and multiple features. To extract the latent structure of multiple features without label information, multi-view learning algorithms have been developed. In this paper, motivated by manifold learning and multi-view Non-negative Matrix Factorization (NMF), we introduce a novel feature extraction method via multi view NMF with local graph regularization, where the inner view relatedness between data is taken into consideration. We propose the matrix factorization objective function by constructing a nearest neighbor graph to integrate local geometrical information of each view and apply two iterative updating rules to effectively solve the optimization problem. In the experiment, we use the extracted feature to cluster several realistic datasets. The experimental results demonstrate the effectiveness of our proposed feature extraction approach.
Nonnegative Matrix Factorization with Local Similarity Learning

Chong Peng,Zhao Kang,Chenglizhao Chen,Qiang Cheng

DOI: https://doi.org/10.48550/arXiv.1907.04150

2019-07-09

Abstract:Existing nonnegative matrix factorization methods focus on learning global structure of the data to construct basis and coefficient matrices, which ignores the local structure that commonly exists among data. In this paper, we propose a new type of nonnegative matrix factorization method, which learns local similarity and clustering in a mutually enhancing way. The learned new representation is more representative in that it better reveals inherent geometric property of the data. Nonlinear expansion is given and efficient multiplicative updates are developed with theoretical convergence guarantees. Extensive experimental results have confirmed the effectiveness of the proposed model.

Machine Learning,Computer Vision and Pattern Recognition
Locality Preserving Nonnegative Matrix Factorization.

Deng Cai,Xiaofei He,Xuanhui Wang,Hujun Bao,Jiawei Han

2009-01-01

Abstract:Matrix factorization techniques have been frequently applied in information processing tasks. Among them, Non-negative Matrix Factorization (NMF) have received considerable attentions due to its psychological and physiological interpretation of naturally occurring data whose representation may be parts-based in human brain. On the other hand, from geometric perspective the data is usually sampled from a low dimensional manifold embedded in high dimensional ambient space. One hopes then to find a compact representation which uncovers the hidden topics and simultaneously respects the intrinsic geometric structure. In this paper, we propose a novel algorithm, called Locality Preserving Non-negative Matrix Factorization (LPNMF), for this purpose. For two data points, we use KL-divergence to evaluate their similarity on the hidden topics. The optimal maps are obtained such that the feature values on hidden topics are restricted to be non-negative and vary smoothly along the geodesics of the data manifold. Our empirical study shows the encouraging results of the proposed algorithm in comparisons to the state-of-the-art algorithms on two large high-dimensional databases.
Robust Structured Nonnegative Matrix Factorization for Image Representation.

Zechao Li,Jinhui Tang,Xiaofei He

DOI: https://doi.org/10.1109/TNNLS.2017.2691725

IF: 14.255

2018-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Dimensionality reduction has attracted increasing attention, because high-dimensional data have arisen naturally in numerous domains in recent years. As one popular dimensionality reduction method, nonnegative matrix factorization (NMF), whose goal is to learn parts-based representations, has been widely studied and applied to various applications. In contrast to the previous approaches, this pape...
A unified statistical approach to non-negative matrix factorization and probabilistic latent semantic indexing

Karthik Devarajan,Guoli Wang,Nader Ebrahimi

DOI: https://doi.org/10.1007/s10994-014-5470-z

2015-04-01

Abstract:Non-negative matrix factorization (NMF) is a powerful machine learning method for decomposing a high-dimensional nonnegative matrix V into the product of two nonnegative matrices, W and H, such that V ∼ W H. It has been shown to have a parts-based, sparse representation of the data. NMF has been successfully applied in a variety of areas such as natural language processing, neuroscience, information retrieval, image processing, speech recognition and computational biology for the analysis and interpretation of large-scale data. There has also been simultaneous development of a related statistical latent class modeling approach, namely, probabilistic latent semantic indexing (PLSI), for analyzing and interpreting co-occurrence count data arising in natural language processing. In this paper, we present a generalized statistical approach to NMF and PLSI based on Renyi's divergence between two non-negative matrices, stemming from the Poisson likelihood. Our approach unifies various competing models and provides a unique theoretical framework for these methods. We propose a unified algorithm for NMF and provide a rigorous proof of monotonicity of multiplicative updates for W and H. In addition, we generalize the relationship between NMF and PLSI within this framework. We demonstrate the applicability and utility of our approach as well as its superior performance relative to existing methods using real-life and simulated document clustering data.
Global and local similarity learning in multi-kernel space for nonnegative matrix factorization

Chong Peng,Xingrong Hou,Yongyong Chen,Zhao Kang,Chenglizhao Chen,Qiang Cheng

DOI: https://doi.org/10.1016/j.knosys.2023.110946

IF: 8.139

2023-11-01

Knowledge-Based Systems

Abstract:Most of existing nonnegative matrix factorization (NMF) methods do not fully exploit global and local similarity information from data. In this paper, we propose a novel local similarity learning approach in the convex NMF framework, which encourages inter-class separability that is desired for clustering. Thus, the new model is capable of enhancing intra-class similarity and inter-class separability with simultaneous global and local learning. Moreover, the model learns the factor matrices in an augmented kernel space, which is a convex combination of pre-defined kernels with auto-learned weights. Thus, the learnings of cluster structure, representation factor matrix, and the optimal kernel mutually enhance each other in a seamlessly integrated model, which leads to informative representation. Multiplicative updating rules are developed with theoretical convergence guarantee. Extensive experimental results have confirmed the effectiveness of the proposed model.

computer science, artificial intelligence
Spatially correlated nonnegative matrix factorization

Xinlei Chen,Cheng Li,Haifeng Liu,Deng Cai

DOI: https://doi.org/10.1016/j.neucom.2013.09.059

IF: 6

2014-01-01

Neurocomputing

Abstract:Low rank approximation of matrices has been frequently applied in information processing tasks, and in recent years, Nonnegative Matrix Factorization (NMF) has received considerable attentions for its straightforward interpretability and superior performance. When applied to image processing, ordinary NMF merely views a p"1xp"2 image as a vector in p"1xp"2-dimensional space and the pixels of the image are considered as independent. It fails to consider the fact that an image displayed in the plane is intrinsically a matrix, and pixels spatially close to each other may probably be correlated. Even though we have p"1xp"2 pixels per image, this spatial correlation suggests that the real number of freedom is far less. In this paper, we introduce a Spatially Correlated Nonnegative Matrix Factorization algorithm, which explicitly models the spatial correlation between neighboring pixels in the parts-based image representation. A multiplicative updating algorithm is also proposed to solve the corresponding optimization problem. Experimental results on benchmark image data sets demonstrate the effectiveness of the proposed method.
Joint Learning of Latent Similarity and Local Embedding for Multi-View Clustering

Aiping Huang,Weiling Chen,Tiesong Zhao,Chang Wen Chen

DOI: https://doi.org/10.1109/tip.2021.3096086

IF: 10.6

2021-01-01

IEEE Transactions on Image Processing

Abstract:Spectral clustering has been an attractive topic in the field of computer vision due to the extensive growth of applications, such as image segmentation, clustering and representation. In this problem, the construction of the similarity matrix is a vital element affecting clustering performance. In this paper, we propose a multi-view joint learning (MVJL) framework to achieve both a reliable similarity matrix and a latent low-dimensional embedding. Specifically, the similarity matrix to be learned is represented as a convex hull of similarity matrices from different views, where the nuclear norm is imposed to capture the principal information of multiple views and improve robustness against noise/outliers. Moreover, an effective low-dimensional representation is obtained by applying local embedding on the similarity matrix, which preserves the local intrinsic structure of data through dimensionality reduction. With these techniques, we formulate the MVJL as a joint optimization problem and derive its mathematical solution with the alternating direction method of multipliers strategy and the proximal gradient descent method. The solution, which consists of a similarity matrix and a low-dimensional representation, is ultimately integrated with spectral clustering or K-means for multi-view clustering. Extensive experimental results on real-world datasets demonstrate that MVJL achieves superior clustering performance over other state-of-the-art methods.
Leveraging maximum entropy and correlation on latent factors for learning representations

Zhicheng He,Jie Liu,Kai Dang,Fuzhen Zhuang,Yalou Huang

DOI: https://doi.org/10.1016/j.neunet.2020.07.027

IF: 7.8

2020-11-01

Neural Networks

Abstract:<p>Many tasks involve learning representations from matrices, and Non-negative Matrix Factorization (NMF) has been widely used due to its excellent interpretability. Through factorization, sample vectors are reconstructed as additive combinations of latent factors, which are represented as non-negative distributions over the raw input features. NMF models are significantly affected by latent factors' distribution characteristics and the correlations among them. And NMF models are faced with the challenge of learning robust latent factor. To this end, we propose to learn representations with an awareness of the semantic quality evaluated from the aspects of intra- and inter-factors. On the one hand, a Maximum Entropy-based function is devised for the intra-factor semantic quality. On the other hand, the semantic uniqueness is evaluated via inter-factor correlation, which reinforces for the aim of semantic compactness. Moreover, we present a novel non-linear NMF framework. The learning algorithm is presented and the convergence is theoretically analyzed and proved. Extensive experimental results on multiple datasets demonstrate that our method can be successfully applied to representative NMF models and boost performances over state-of-the-art models.</p>

computer science, artificial intelligence,neurosciences
Learnable Similarity and Dissimilarity Guided Symmetric Non-Negative Matrix Factorization

Wenlong Lyu,Yuheng Jia

2024-12-05

Abstract:Symmetric nonnegative matrix factorization (SymNMF) is a powerful tool for clustering, which typically uses the $k$-nearest neighbor ($k$-NN) method to construct similarity matrix. However, $k$-NN may mislead clustering since the neighbors may belong to different clusters, and its reliability generally decreases as $k$ grows. In this paper, we construct the similarity matrix as a weighted $k$-NN graph with learnable weight that reflects the reliability of each $k$-th NN. This approach reduces the search space of the similarity matrix learning to $n - 1$ dimension, as opposed to the $\mathcal{O}(n^2)$ dimension of existing methods, where $n$ represents the number of samples. Moreover, to obtain a discriminative similarity matrix, we introduce a dissimilarity matrix with a dual structure of the similarity matrix, and propose a new form of orthogonality regularization with discussions on its geometric interpretation and numerical stability. An efficient alternative optimization algorithm is designed to solve the proposed model, with theoretically guarantee that the variables converge to a stationary point that satisfies the KKT conditions. The advantage of the proposed model is demonstrated by the comparison with nine state-of-the-art clustering methods on eight datasets. The code is available at \url{<a class="link-external link-https" href="https://github.com/lwl-learning/LSDGSymNMF" rel="external noopener nofollow">this https URL</a>}.

Machine Learning
Local Learning Regularized Nonnegative Matrix Factorization.

Quanquan Gu,Jie Zhou

2009-01-01

Abstract:Nonnegative Matrix Factorization (NMF) has been widely used in machine learning and data mining. It aims to find two nonnegative matrices whose product can well approximate the nonnegative data matrix, which naturally lead to parts-based representation. In this paper, we present a local learning regularized nonnegative matrix factorization (LLNMF) for clustering. It imposes an additional constraint on NMF that the cluster label of each point can be predicted by the points in its neighborhood. This constraint encodes both the discriminative information and the geometric structure, and is good at clustering data on manifold. An iterative multiplicative updating algorithm is proposed to optimize the objective, and its convergence is guaranteed theoretically. Experiments on many benchmark data sets demonstrate that the proposed method outperforms NMF as well as many state of the art clustering methods.
Spatially correlated nonnegative matrix factorization for image analysis

Xinlei Chen,Cheng Li,Deng Cai

DOI: https://doi.org/10.1007/978-3-642-36669-7_19

2013-01-01

Abstract:Low rank approximation of matrices has been frequently applied in information processing tasks, and in recent years, Nonnegative Matrix Factorization (NMF) has received considerable attentions for its straightforward interpretability and superior performance. When applied to image processing, ordinary NMF merely views a p1×p2 image as a vector in p1×p2-dimensional space and the pixels of the image are considered as independent. It fails to consider the fact that an image displayed in the plane is intrinsically a matrix, and pixels spatially close to each other may probably be correlated. Even though we have p1×p2 pixels per image, this spatial correlation suggests the real number of freedom is far less. In this paper, we introduce a Spatially Correlated Nonnegative Matrix Factorization algorithm, which explicitly models the spatial correlation between neighboring pixels in the parts-based image representation. A multiplicative updating algorithm is also proposed to solve the corresponding optimization problem. Experimental results on benchmark image data sets demonstrate the effectiveness of the proposed method.
Multi-label feature selection via similarity constraints with non-negative matrix factorization

Zhuoxin He,Yaojin Lin,Zilong Lin,Chenxi Wang

DOI: https://doi.org/10.1016/j.knosys.2024.111948

IF: 8.139

2024-05-23

Knowledge-Based Systems

Abstract:Feature selection plays a key role in preprocessing, effectively addressing the curse of dimensionality in multi-label learning. While current approaches commonly utilize feature or label similarity to construct the weight matrix, typically through manifold learning regularization, there has been a dearth of progress in developing similarity-constrained regularization terms for multi-label feature selection. To address this gap, this paper conducts a comprehensive investigation and proposes a novel similarity constraint leveraging non-negative matrix factorization techniques. Subsequently, a new algorithm is introduced termed Multi-label Feature Selection via Similarity Constraints with Non-negative Matrix Factorization (SCNMF). Initially, the Gaussian similarity matrix among features is computed and factorized into a weight matrix using non-negative matrix factorization. Subsequently, the Cosine distance matrix among labels is computed to constrain the weight matrix. Finally, an objective function is formulated based on the aforementioned constraint mechanisms and iteratively optimized. Extensive experiments conducted across more than 10 multi-label datasets demonstrate the superior performance of our proposed approach. The source code is accessible at the URL: https://github.com/BIGOatMNNU/SCNMF .

computer science, artificial intelligence
Learning Parts-Based and Global Representation for Image Classification.

Yuwu Lu,Zhihui Lai,Xuelong Li,David Zhang,Wai Keung Wong,Chun Yuan

DOI: https://doi.org/10.1109/tcsvt.2017.2749980

IF: 5.859

2018-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Nonnegative matrix factorization (NMF), known as a famous matrix factorization technique, has been widely used in pattern recognition and computer vision. NMF represents the input data matrix as a product of two nonnegative factors. As NMF is based on the Euclidean distance, which is sensitive to noise or errors in the data, some robust NMF methods are proposed. Mainly focusing on parts-based representation, these robust NMF methods often neglect global representation of data. In fact, the global geometry information of data is more robust than the local information about the noisy data in terms of image classification. In order to effectively improve the robustness of NMF and learn part-based and global representation of the data, a novel method low-rank nonnegative factorization (LRNF) is proposed in this paper. First, we assume that the data are grossly corrupted, and the $L_{1} $ norm is used as a sparse constraint on the assumed noise matrix. Then, LRNF learns a low-rank matrix with the global representation ability. Finally, we make a nonnegative factorization of the learned low-rank matrix. We can obtain a base matrix, which preserves locality and globality properties of the data in the meantime. Extensive experiments have been conducted on nine real-world image databases to verify the performance of the proposed LRNF method by comparing with the state-of-the-art algorithms on robust dimensionality reduction.
Hyperspectral Unmixing Using Robust Deep Nonnegative Matrix Factorization

Risheng Huang,Huiyun Jiao,Xiaorun Li,Shuhan Chen,Chaoqun Xia

DOI: https://doi.org/10.3390/rs15112900

IF: 5

2023-01-01

Remote Sensing

Abstract:Nonnegative matrix factorization (NMF) and its numerous variants have been extensively studied and used in hyperspectral unmixing (HU). With the aid of the designed deep structure, deep NMF-based methods demonstrate advantages in exploring the hierarchical features of complex data. However, a noise corruption problem commonly exists in hyperspectral data and severely degrades the unmixing performance of deep NMF-based methods when applied to HU. In this study, we propose an ℓ2,1 norm-based robust deep nonnegative matrix factorization (ℓ2,1-RDNMF) for HU, which incorporates an ℓ2,1 norm into the two stages of the deep structure to achieve robustness. The multiplicative updating rules of ℓ2,1-RDNMF are efficiently learned and provided. The efficiency of the presented method is verified in experiments using both synthetic and genuine data.
Robust discriminative non-negative matrix factorization.

Ruiqing Zhang,Zhenfang Hu,Gang Pan,Yueming Wang

DOI: https://doi.org/10.1016/j.neucom.2015.07.032

IF: 6

2016-01-01

Neurocomputing

Abstract:Traditional non-negative matrix factorization (NMF) is an unsupervised method that represents non-negative data by a part-based dictionary and non-negative codes. Recently, the unsupervised NMF has been extended to discriminative ones for classification problems. However, these discriminative methods may become inefficient when outliers are presented in the data, e.g. mislabeled samples, because the outliers usually deviate from the normal samples in one class and would perturb the discriminative dictionary. In this paper, we propose a novel method, called robust discriminative non-negative matrix factorization (RDNMF), to reduce the effect of outliers and improve the discriminative strength. The RDNMF learns a non-negative dictionary for each class, and each dictionary contains two parts: a discriminative part and an outlier part. The discriminative parts are obtained by minimizing the cosine similarity between classes. The codes on the outlier part are required to be sparse so that most outliers can be modeled by this part, without large influence over the discriminative part.The final dictionary is obtained by concatenating the discriminative parts of all classes, and the non-negative codes for each sample, as well as test sample, are obtained by coding with this dictionary. Experimental comparisons with existing dictionary learning methods on MNIST, PIE, Yale B and ORL demonstrate the effectiveness and robustness of our approach.
Non-Negative Matrix Factorization with Constraints.

Haifeng Liu,Zhaohui Wu

DOI: https://doi.org/10.1609/aaai.v24i1.7681

2010-01-01

Abstract:Non-negative matrix factorization (NMF), as a useful decomposition method for multivariate data, has been widely used in pattern recognition, information retrieval and computer vision. NMF is an effective algorithm to find the latent structure of the data and leads to a parts-based representation. However, NMF is essentially an unsupervised method and can not make use of label information. In this paper, we propose a novel semi-supervised matrix decomposition method, called Constrained Non-negative Matrix Factorization, which takes the label information as additional constraints. Specifically, we require that the data points sharing the same label have the same coordinate in the new representation space. This way, the learned representations can have more discriminating power. We demonstrate the effectiveness of this novel algorithm through a set of evaluations on real world applications.
Nonnegative Matrix Factorization for Multimodality Data from Multi-source Domain

Xu Tan,Shuai Ma,Jian Shao,Yueting Zhuang,Fei Wu

DOI: https://doi.org/10.1109/IIH-MSP.2012.87

2012-01-01

Abstract:With the growing popularity of social tagging, more and more images are annotated by users on web sites(e.g., Flickr, Blogspace and Youtube). Since the tags annotated by users are often noisy, ambiguous, and subjective, it is beneficial to fully utilize the multiview information and borrow strength from multiple data sources to boost the performance of image annotation and tag-based image retrieval. Therefore, the appropriate integration and utilization of complimentary cues from multiple modality in multiple data sources is an important research topic. Inspired by the recent advances of multiview learning and shared subspace learning, this paper proposes an approach, namely Multimodality Multi-source Nonnegative Matrix Factorization (M2NMF) to learn a shared and corresponding individual structures from multimodality data via nonnegative matrix factorization. The experimental results demonstrate the feasibility and effectiveness of the proposed approach.
Non-negative Local Coordinate Factorization for Image Representation.

Yan Chen,Jiemi Zhang,Deng Cai,Wei Liu,Xiaofei He

DOI: https://doi.org/10.1109/tip.2012.2224357

2013-01-01

Abstract:Recently, nonnegative matrix factorization (NMF) has become increasingly popular for feature extraction in computer vision and pattern recognition. NMF seeks two nonnegative matrices whose product can best approximate the original matrix. The nonnegativity constraints lead to sparse parts-based representations that can be more robust than nonsparse global features. To obtain more accurate control over the sparseness, in this paper, we propose a novel method called nonnegative local coordinate factorization (NLCF) for feature extraction. NLCF adds a local coordinate constraint into the standard NMF objective function. Specifically, we require that the learned basis vectors be as close to the original data points as possible. In this way, each data point can be represented by a linear combination of only a few nearby basis vectors, which naturally leads to sparse representation. Extensive experimental results suggest that the proposed approach provides a better representation and achieves higher accuracy in image clustering.
Convergence Analysis of Graph Regularized Non-Negative Matrix Factorization

Shangming Yang,Zhang Yi,Mao Ye,Xiaofei He

DOI: https://doi.org/10.1109/tkde.2013.98

IF: 9.235

2013-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Graph regularized non-negative matrix factorization (NMF) algorithms can be applied to information retrieval, image processing, and pattern recognition. However, challenge that still remains is to prove the convergence of this class of learning algorithms since the geometrical structure of the data space is considered. This paper presents the convergence properties of the graph regularized NMF learning algorithms. In the analysis, we focus on the study of Euclidian distance based algorithms. The structures of the fixed points are presented. The non-divergence of the learning algorithms is analyzed by constructing invariant sets for update rules. Based on Lyapunov indirect method, the stability of the algorithms is discussed in detail. The analysis shows that this class of NMF algorithms can converge to their fixed points under some given conditions. In the simulations, theoretical results presented in the paper are confirmed. For different initializations and data sets, variations of cost functions and decomposition data in the learning are presented to show the convergence features of the discussed NMF update rules, and the convergence speed of the algorithms is also investigated.

Learning latent features by nonnegative matrix factorization combining similarity judgments

Feature Extraction Via Multi-View Non-Negative Matrix Factorization with Local Graph Regularization.

Nonnegative Matrix Factorization with Local Similarity Learning

Locality Preserving Nonnegative Matrix Factorization.

Robust Structured Nonnegative Matrix Factorization for Image Representation.

A unified statistical approach to non-negative matrix factorization and probabilistic latent semantic indexing

Global and local similarity learning in multi-kernel space for nonnegative matrix factorization

Spatially correlated nonnegative matrix factorization

Joint Learning of Latent Similarity and Local Embedding for Multi-View Clustering

Leveraging maximum entropy and correlation on latent factors for learning representations

Learnable Similarity and Dissimilarity Guided Symmetric Non-Negative Matrix Factorization

Local Learning Regularized Nonnegative Matrix Factorization.

Spatially correlated nonnegative matrix factorization for image analysis

Multi-label feature selection via similarity constraints with non-negative matrix factorization

Learning Parts-Based and Global Representation for Image Classification.

Hyperspectral Unmixing Using Robust Deep Nonnegative Matrix Factorization

Robust discriminative non-negative matrix factorization.

Non-Negative Matrix Factorization with Constraints.

Nonnegative Matrix Factorization for Multimodality Data from Multi-source Domain

Non-negative Local Coordinate Factorization for Image Representation.

Convergence Analysis of Graph Regularized Non-Negative Matrix Factorization