Nonsmooth rank-one symmetric matrix factorization landscape

Cédric Josz,Lexiao Lai

2024-10-23

Abstract:We consider nonsmooth rank-one symmetric matrix factorization. It has no spurious second-order stationary points.

Optimization and Control

What problem does this paper attempt to address?

$\ell_1$-norm rank-one symmetric matrix factorization has no spurious second-order stationary points

Jiewen Guan,Anthony Man-Cho So

2024-10-07

Abstract:This paper studies the nonsmooth optimization landscape of the $\ell_1$-norm rank-one symmetric matrix factorization problem using tools from second-order variational analysis. Specifically, as the main finding of this paper, we show that any second-order stationary point (and thus local minimizer) of the problem is actually globally optimal. Besides, some other results concerning the landscape of the problem, such as a complete characterization of the set of stationary points, are also developed, which should be interesting in their own rights. Furthermore, with the above theories, we revisit existing results on the generic minimizing behavior of simple algorithms for nonsmooth optimization and showcase the potential risk of their applications to our problem through several examples. Our techniques can potentially be applied to analyze the optimization landscapes of a variety of other more sophisticated nonsmooth learning problems, such as robust low-rank matrix recovery.

Optimization and Control,Machine Learning
Nonconvex Factorization and Manifold Formulations are Almost Equivalent in Low-rank Matrix Optimization

Yuetian Luo,Xudong Li,Anru R. Zhang

2024-08-13

Abstract:In this paper, we consider the geometric landscape connection of the widely studied manifold and factorization formulations in low-rank positive semidefinite (PSD) and general matrix optimization. We establish a sandwich relation on the spectrum of Riemannian and Euclidean Hessians at first-order stationary points (FOSPs). As a result of that, we obtain an equivalence on the set of FOSPs, second-order stationary points (SOSPs) and strict saddles between the manifold and the factorization formulations. In addition, we show the sandwich relation can be used to transfer more quantitative geometric properties from one formulation to another. Similarities and differences in the landscape connection under the PSD case and the general case are discussed. To the best of our knowledge, this is the first geometric landscape connection between the manifold and the factorization formulations for handling rank constraints, and it provides a geometric explanation for the similar empirical performance of factorization and manifold approaches in low-rank matrix optimization observed in the literature. In the general low-rank matrix optimization, the landscape connection of two factorization formulations (unregularized and regularized ones) is also provided. By applying these geometric landscape connections, in particular, the sandwich relation, we are able to solve unanswered questions in literature and establish stronger results in the applications on geometric analysis of phase retrieval, well-conditioned low-rank matrix optimization, and the role of regularization in factorization arising from machine learning and signal processing.

Optimization and Control,Information Theory,Machine Learning,Signal Processing
Robust Structured Nonnegative Matrix Factorization for Image Representation.

Zechao Li,Jinhui Tang,Xiaofei He

DOI: https://doi.org/10.1109/TNNLS.2017.2691725

IF: 14.255

2018-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Dimensionality reduction has attracted increasing attention, because high-dimensional data have arisen naturally in numerous domains in recent years. As one popular dimensionality reduction method, nonnegative matrix factorization (NMF), whose goal is to learn parts-based representations, has been widely studied and applied to various applications. In contrast to the previous approaches, this pape...
Stability properties of gradient flow dynamics for the symmetric low-rank matrix factorization problem

Hesameddin Mohammadi,Mohammad Tinati,Stephen Tu,Mahdi Soltanolkotabi,Mihailo R. Jovanović

2024-11-25

Abstract:The symmetric low-rank matrix factorization serves as a building block in many learning tasks, including matrix recovery and training of neural networks. However, despite a flurry of recent research, the dynamics of its training via non-convex factorized gradient-descent-type methods is not fully understood especially in the over-parameterized regime where the fitted rank is higher than the true rank of the target matrix. To overcome this challenge, we characterize equilibrium points of the gradient flow dynamics and examine their local and global stability properties. To facilitate a precise global analysis, we introduce a nonlinear change of variables that brings the dynamics into a cascade connection of three subsystems whose structure is simpler than the structure of the original system. We demonstrate that the Schur complement to a principal eigenspace of the target matrix is governed by an autonomous system that is decoupled from the rest of the dynamics. In the over-parameterized regime, we show that this Schur complement vanishes at an $O(1/t)$ rate, thereby capturing the slow dynamics that arises from excess parameters. We utilize a Lyapunov-based approach to establish exponential convergence of the other two subsystems. By decoupling the fast and slow parts of the dynamics, we offer new insight into the shape of the trajectories associated with local search algorithms and provide a complete characterization of the equilibrium points and their global stability properties. Such an analysis via nonlinear control techniques may prove useful in several related over-parameterized problems.

Machine Learning,Systems and Control,Dynamical Systems,Optimization and Control
Adversarially-Trained Nonnegative Matrix Factorization

Ting Cai,Vincent Y. F. Tan,Cédric Févotte

DOI: https://doi.org/10.1109/LSP.2021.3092231

2021-06-22

Abstract:We consider an adversarially-trained version of the nonnegative matrix factorization, a popular latent dimensionality reduction technique. In our formulation, an attacker adds an arbitrary matrix of bounded norm to the given data matrix. We design efficient algorithms inspired by adversarial training to optimize for dictionary and coefficient matrices with enhanced generalization abilities. Extensive simulations on synthetic and benchmark datasets demonstrate the superior predictive performance on matrix completion tasks of our proposed method compared to state-of-the-art competitors, including other variants of adversarial nonnegative matrix factorization.

Machine Learning,Signal Processing
Rethinking Symmetric Matrix Factorization: A More General and Better Clustering Perspective

Mengyuan Zhang,Kai Liu

2023-11-04

Abstract:Nonnegative matrix factorization (NMF) is widely used for clustering with strong interpretability. Among general NMF problems, symmetric NMF is a special one that plays an important role in graph clustering where each element measures the similarity between data points. Most existing symmetric NMF algorithms require factor matrices to be nonnegative, and only focus on minimizing the gap between similarity matrix and its approximation for clustering, without giving a consideration to other potential regularization terms which can yield better clustering. In this paper, we explore factorizing a symmetric matrix that does not have to be nonnegative, presenting an efficient factorization algorithm with a regularization term to boost the clustering performance. Moreover, a more general framework is proposed to solve symmetric matrix factorization problems with different constraints on the factor matrices.

Machine Learning,Artificial Intelligence
Butterfly factorization by algorithmic identification of rank-one blocks

Léon Zheng,Gilles Puy,Elisa Riccietti,Patrick Pérez,Rémi Gribonval

2023-07-03

Abstract:Many matrices associated with fast transforms posess a certain low-rank property characterized by the existence of several block partitionings of the matrix, where each block is of low rank. Provided that these partitionings are known, there exist algorithms, called butterfly factorization algorithms, that approximate the matrix into a product of sparse factors, thus enabling a rapid evaluation of the associated linear operator. This paper proposes a new method to identify algebraically these block partitionings for a matrix admitting a butterfly factorization, without any analytical assumption on its entries.

Numerical Analysis
Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization

Liwei Jiang,Yudong Chen,Lijun Ding

DOI: https://doi.org/10.48550/arXiv.2203.02839

2022-09-16

Abstract:We study the asymmetric matrix factorization problem under a natural nonconvex formulation with arbitrary overparametrization. The model-free setting is considered, with minimal assumption on the rank or singular values of the observed matrix, where the global optima provably overfit. We show that vanilla gradient descent with small random initialization sequentially recovers the principal components of the observed matrix. Consequently, when equipped with proper early stopping, gradient descent produces the best low-rank approximation of the observed matrix without explicit regularization. We provide a sharp characterization of the relationship between the approximation error, iteration complexity, initialization size and stepsize. Our complexity bound is almost dimension-free and depends logarithmically on the approximation error, with significantly more lenient requirements on the stepsize and initialization compared to prior work. Our theoretical results provide accurate prediction for the behavior gradient descent, showing good agreement with numerical experiments.

Machine Learning,Optimization and Control
No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis

Rong Ge,Chi Jin,Yi Zheng

DOI: https://doi.org/10.48550/arXiv.1704.00708

2017-04-04

Abstract:In this paper we develop a new framework that captures the common landscape underlying the common non-convex low-rank matrix problems including matrix sensing, matrix completion and robust PCA. In particular, we show for all above problems (including asymmetric cases): 1) all local minima are also globally optimal; 2) no high-order saddle points exists. These results explain why simple algorithms such as stochastic gradient descent have global converge, and efficiently optimize these non-convex objective functions in practice. Our framework connects and simplifies the existing analyses on optimization landscapes for matrix sensing and symmetric matrix completion. The framework naturally leads to new results for asymmetric matrix completion and robust PCA.

Machine Learning,Optimization and Control
Notes on Low-rank Matrix Factorization

Yuan Lu,Jie Yang

DOI: https://doi.org/10.48550/arXiv.1507.00333

2016-05-06

Abstract:Low-rank matrix factorization (MF) is an important technique in data science. The key idea of MF is that there exists latent structures in the data, by uncovering which we could obtain a compressed representation of the data. By factorizing an original matrix to low-rank matrices, MF provides a unified method for dimension reduction, clustering, and matrix completion. In this article we review several important variants of MF, including: Basic MF, Non-negative MF, Orthogonal non-negative MF. As can be told from their names, non-negative MF and orthogonal non-negative MF are variants of basic MF with non-negativity and/or orthogonality constraints. Such constraints are useful in specific senarios. In the first part of this article, we introduce, for each of these models, the application scenarios, the distinctive properties, and the optimizing method. By properly adapting MF, we can go beyond the problem of clustering and matrix completion. In the second part of this article, we will extend MF to sparse matrix compeletion, enhance matrix compeletion using various regularization methods, and make use of MF for (semi-)supervised learning by introducing latent space reinforcement and transformation. We will see that MF is not only a useful model but also as a flexible framework that is applicable for various prediction problems.

Numerical Analysis,Information Retrieval,Machine Learning
Accelerated Structured Matrix Factorization

Lorenzo Schiavon,Bernardo Nipoti,Antonio Canale

DOI: https://doi.org/10.1080/10618600.2023.2301072

2024-02-09

Journal of Computational and Graphical Statistics

Abstract:Matrix factorization exploits the idea that, in complex high-dimensional data, the actual signal typically lies in lower-dimensional structures. These lower dimensional objects provide useful insights, with interpretation favored by sparse structures. Sparsity, in addition, is beneficial in terms of regularization and, thus, to avoid over-fitting. By exploiting Bayesian shrinkage priors, we devise a computationally convenient approach for high-dimensional matrix factorization. The dependence between row and column entities is modeled by inducing flexible sparse patterns within factors. The availability of external information is accounted for in such a way that structures are allowed while not imposed. Inspired by boosting algorithms, we pair the proposed approach with a numerical strategy relying on a sequential inclusion and estimation of low-rank contributions, with a data-driven stopping rule. Practical advantages of the proposed approach are demonstrated by means of a simulation study and the analysis of soccer heatmaps obtained from new generation tracking data. Supplementary materials for this article are available online.

statistics & probability
A multiscale cavity method for sublinear-rank symmetric matrix factorization

Jean Barbier,Justin Ko,Anas A. Rahman

2024-03-12

Abstract:We consider a statistical model for symmetric matrix factorization with additive Gaussian noise in the high-dimensional regime where the rank $M$ of the signal matrix to infer scales with its size $N$ as $M = o(N^{1/10})$. Allowing for a $N$-dependent rank offers new challenges and requires new methods. Working in the Bayesian-optimal setting, we show that whenever the signal has i.i.d. entries the limiting mutual information between signal and data is given by a variational formula involving a rank-one replica symmetric potential. In other words, from the information-theoretic perspective, the case of a (slowly) growing rank is the same as when $M = 1$ (namely, the standard spiked Wigner model). The proof is primarily based on a novel multiscale cavity method allowing for growing rank along with some information-theoretic identities on worst noise for the Gaussian vector channel. We believe that the cavity method developed here will play a role in the analysis of a broader class of inference and spin models where the degrees of freedom are large arrays instead of vectors.

Information Theory,Disordered Systems and Neural Networks,Mathematical Physics,Statistics Theory
A Symmetric Rank-one Quasi Newton Method for Non-negative Matrix Factorization

Shu-Zhen Lai,Hou-Biao Li,Zu-Tao Zhang

DOI: https://doi.org/10.48550/arXiv.1305.5829

2013-05-25

Abstract:As we all known, the nonnegative matrix factorization (NMF) is a dimension reduction method that has been widely used in image processing, text compressing and signal processing etc. In this paper, an algorithm for nonnegative matrix approximation is proposed. This method mainly bases on the active set and the quasi-Newton type algorithm, by using the symmetric rank-one and negative curvature direction technologies to approximate the Hessian matrix. Our method improves the recent results of those methods in [Pattern Recognition, 45(2012)3557-3565; SIAM J. Sci. Comput., 33(6)(2011)3261-3281; Neural Computation, 19(10)(2007)2756-2779, etc.]. Moreover, the object function decreases faster than many other NMF methods. In addition, some numerical experiments are presented in the synthetic data, imaging processing and text clustering. By comparing with the other six nonnegative matrix approximation methods, our experiments confirm to our analysis.

Numerical Analysis,Machine Learning
Non-negative Matrix-Set Factorization

Le Li,Yu-Jin Zhang

DOI: https://doi.org/10.1109/icig.2007.103

2009-01-01

Abstract:Non-negative matrix factorization (NMF) is a recently developed, biologically inspired method for nonlinearly finding purely additive, sparse, linear, low-dimension representations of non-negative multivariate data to consequently make latent structure, feature or pattern in the data clear. Although it has been successfully applied in several research fields, it is confronted with three main problems, unsatisfactory accuracy, bad generality and high computational load, while the processed data appear as matrices. In this paper, a new method coined non-negative matrix-set factorization (NMSF) is developed to overcome the problems and an efficient, strictly monotonically convergent algorithm of NMSF is put forward. As opposed to NMF, NMSF directly processes original data matrices rather than vectorization results of them. Theoretical analysis and experimental results show that NMSF has higher accuracy, better generality and lower computational load than NMF.
Non-negative Matrix Factorization on Manifold

Deng Cai,Xiaofei He,Xiaoyun Wu,Jiawei Han

DOI: https://doi.org/10.1109/ICDM.2008.57

2008-01-01

Abstract:Recently non-negative matrix factorization (NMF) has received a lot of attentions in information retrieval, computer vision and pattern recognition. NMF aims to find two non-negative matrices whose product can well approximate the original matrix. The sizes of these two matrices are usually smaller than the original matrix. This results in a compressed version of the original data matrix. The solution of NMF yields a natural parts-based representation for the data. When NMF is applied for data representation, a major disadvantage is that it fails to consider the geometric structure in the data. In this paper, we develop a graph based approach for parts-based data representation in order to overcome this limitation. We construct an affinity graph to encode the geometrical information and seek a matrix factorization which respects the graph structure. We demonstrate the success of this novel algorithm by applying it on real world problems.
Structured Low-Rank Matrix Factorization: Global Optimality, Algorithms, and Applications

Benjamin D. Haeffele,René Vidal,Rene Vidal

DOI: https://doi.org/10.1109/tpami.2019.2900306

IF: 23.6

2020-06-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Convex formulations of low-rank matrix factorization problems have received considerable attention in machine learning. However, such formulations often require solving for a matrix of the size of the data matrix, making it challenging to apply them to large scale datasets. Moreover, in many applications the data can display structures beyond simply being low-rank, e.g., images and videos present complex spatio-temporal structures that are largely ignored by standard low-rank methods. In this paper we study a matrix factorization technique that is suitable for large datasets and captures additional structure in the factors by using a particular form of regularization that includes well-known regularizers such as total variation and the nuclear norm as particular cases. Although the resulting optimization problem is non-convex, we show that if the size of the factors is large enough, under certain conditions, any local minimizer for the factors yields a global minimizer. A few practical algorithms are also provided to solve the matrix factorization problem, and bounds on the distance from a given approximate solution of the optimization problem to the global optimum are derived. Examples in neural calcium imaging video segmentation and hyperspectral compressed recovery show the advantages of our approach on high-dimensional datasets.

computer science, artificial intelligence,engineering, electrical & electronic
Joint Tensor Factorization and Outlying Slab Suppression With Applications

Xiao Fu,Kejun Huang,Wing-Kin Ma,Nicholas D. Sidiropoulos,Rasmus Bro

DOI: https://doi.org/10.1109/tsp.2015.2469642

IF: 4.875

2015-12-01

IEEE Transactions on Signal Processing

Abstract:We consider factoring low-rank tensors in the presence of outlying slabs. This problem is important in practice, because data collected in many real-world applications, such as speech, fluorescence, and some social network data, fit this paradigm. Prior work tackles this problem by iteratively selecting a fixed number of slabs and fitting, a procedure which may not converge. We formulate this problem from a group-sparsity promoting point of view, and propose an alternating optimization framework to handle the corresponding $\ell_{p}$ $(0

engineering, electrical & electronic
Sparse Factorization of Large Square Matrices

Ruslan Khalitov,Tong Yu,Lei Cheng,Zhirong Yang

DOI: https://doi.org/10.48550/arXiv.2109.08184

2021-09-17

Abstract:Square matrices appear in many machine learning problems and models. Optimization over a large square matrix is expensive in memory and in time. Therefore an economic approximation is needed. Conventional approximation approaches factorize the square matrix into a number matrices of much lower ranks. However, the low-rank constraint is a performance bottleneck if the approximated matrix is intrinsically high-rank or close to full rank. In this paper, we propose to approximate a large square matrix with a product of sparse full-rank matrices. In the approximation, our method needs only $N(\log N)^2$ non-zero numbers for an $N\times N$ full matrix. We present both non-parametric and parametric ways to find the factorization. In the former, we learn the factorizing matrices directly, and in the latter, we train neural networks to map input data to the non-zero matrix entries. The sparse factorization method is tested for a variety of synthetic and real-world square matrices. The experimental results demonstrate that our method gives a better approximation when the approximated matrix is sparse and high-rank. Based on this finding, we use our parametric method as a scalable attention architecture that performs strongly in learning tasks for long sequential data and defeats Transformer and its several variants.

Machine Learning
Approximation method to rank-one binary matrix factorization

Zhongshun Shi,Longfei Wang,Leyuan Shi

DOI: https://doi.org/10.1109/CoASE.2014.6899417

2014-01-01

CASE

Abstract:In this paper, we study the rank-one binary matrix factorization problem, which plays a key role in analyzing the high dimensional discrete-attributed datasets. We first transform this problem into an equivalent binary quadratic programming, and then a mixed-integer linear formulation is obtained using linearization technique. A characterization on the extreme points is given for the polyhedron of the linearized formulation. To the best of our knowledge, this is the first time to study the extreme points property of the rank-one binary matrix factorization. Based on this property, a LP rounding based 2-approximation algorithm is designed for this problem. Numerical results illustrate the efficiency of the proposed approximation algorithm.
Sparse Matrix Factorization with L2, 1 Norm for Matrix Completion

Xiaobo Jin,Jianyu Miao,Qiufeng Wang,Guanggang Geng,Kaizhu Huang

DOI: https://doi.org/10.1016/j.patcog.2022.108655

IF: 8

2022-01-01

Pattern Recognition

Abstract:Matrix factorization is a popular matrix completion method, however, it is difficult to determine the ranks of the factor matrices. We propose two new sparse matrix factorization methods with l(2,1) norm to explicitly force the row sparseness of the factor matrices, where the rank of the factor matrices is adaptively controlled by the regularization coefficient. We further theoretically prove the convergence property of our algorithms. The experimental results on the simulation and the benchmark datasets show that our methods achieve superior performance than its counterparts. Moreover our proposed methods can attain comparable performance with the deep learning-based matrix completion methods. (C) 2022 Elsevier Ltd. All rights reserved.

Nonsmooth rank-one symmetric matrix factorization landscape

$\ell_1$-norm rank-one symmetric matrix factorization has no spurious second-order stationary points

Nonconvex Factorization and Manifold Formulations are Almost Equivalent in Low-rank Matrix Optimization

Robust Structured Nonnegative Matrix Factorization for Image Representation.

Stability properties of gradient flow dynamics for the symmetric low-rank matrix factorization problem

Adversarially-Trained Nonnegative Matrix Factorization

Rethinking Symmetric Matrix Factorization: A More General and Better Clustering Perspective

Butterfly factorization by algorithmic identification of rank-one blocks

Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization

No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis

Notes on Low-rank Matrix Factorization

Accelerated Structured Matrix Factorization

A multiscale cavity method for sublinear-rank symmetric matrix factorization

A Symmetric Rank-one Quasi Newton Method for Non-negative Matrix Factorization

Non-negative Matrix-Set Factorization

Non-negative Matrix Factorization on Manifold

Structured Low-Rank Matrix Factorization: Global Optimality, Algorithms, and Applications

Joint Tensor Factorization and Outlying Slab Suppression With Applications

Sparse Factorization of Large Square Matrices

Approximation method to rank-one binary matrix factorization

Sparse Matrix Factorization with L2, 1 Norm for Matrix Completion