“Self-Paced Learning for Matrix Factorization”: Supplementary Material

Qian Zhao,Deyu Meng,Lu Jiang,Qi Xie,Zongben Xu,Alexander G Hauptmann

2015-01-01

Abstract:In this supplementary material, we give the proof of Theorem 1 in the maintext.

What problem does this paper attempt to address?

Self-Paced Learning for Matrix Factorization

Qian Zhao,Deyu Meng,Lu Jiang,Qi Xie,Zongben Xu,Alexander G. Hauptmann

DOI: https://doi.org/10.1609/aaai.v29i1.9584

2015-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Matrix factorization (MF) has been attracting much attention due to its wide applications. However, since MF models are generally non-convex, most of the existing methods are easily stuck into bad local minima, especially in the presence of outliers and missing data. To alleviate this deficiency, in this study we present a new MF learning methodology by gradually including matrix elements into MF training from easy to complex. This corresponds to a recently proposed learning fashion called self-paced learning (SPL), which has been demonstrated to be beneficial in avoiding bad local minima. We also generalize the conventional binary (hard) weighting scheme for SPL to a more effective real-valued (soft) weighting manner. The effectiveness of the proposed self-paced MF method is substantiated by a series of experiments on synthetic, structure from motion and background subtraction data.
Additional Online Supplement for “Inference for Approximate Factor Models: Random Missing and Cross Validation”

Liangjun Su,Ke Miao,Sainan Jin

2019-01-01

Abstract:Proof of Lemma A.1. From the principal component analysis (PCA), we have the identity ( NT q̃2 )−1 X̃X̃ ′F̃ = F̃ D̃. Pre-multiplying both sides by T−1F̃ ′ and using the normalization T−1F̃ ′F̃ = IR yield T−1F̃ ′ ( NT q̃2 )−1 X̃X̃ ′F̃ = D̃. The lemma follows providedD =plimD̃, which we show below. Noting that X̃ = (F 0Λ0′ + ε) ◦G, we have D̃ = T−1F̃ ′ ( NT q̃ )−1 X̃X̃ ′F̃ = T−1F̃ ′ ( NT q̃ )−1 [ (F 0Λ0′ + ε) ◦G ] [ (F 0Λ0′ + ε) ◦G ]′ F̃ = T−1F̃ ′ ( NT q̃ )−1 [ (F 0Λ0′) ◦G ] [ (F 0Λ0′) ◦G ]′ F̃ + T−1F̃ ′ ( NT q̃ )−1 (ε ◦G) (ε ◦G)′F̃ +T−1F̃ ′ ( NT q̃ )−1 [ (F 0Λ0′) ◦G ] (ε ◦G)′F̃ + T−1F̃ ′ ( NT q̃ )−1 (ε ◦G)′ [ (F 0Λ0′) ◦G ]′ F̃
Quadratic Matrix Factorization With Applications to Manifold Learning

Zheng Zhai,Hengchao Chen,Qiang Sun

DOI: https://doi.org/10.1109/tpami.2024.3380568

IF: 23.6

2024-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Matrix factorization is a popular framework for modeling low-rank data matrices. Motivated by manifold learning problems, this paper proposes a quadratic matrix factorization (QMF) framework to learn the curved manifold on which the dataset lies. Unlike local linear methods such as the local principal component analysis, QMF can better exploit the curved structure of the underlying manifold. Algorithmically, we propose an alternating minimization algorithm to optimize QMF and establish its theoretical convergence properties. Moreover, to avoid possible over-fitting, we then propose a regularized QMF algorithm and discuss how to tune its regularization parameter. Finally, we elaborate how to apply the regularized QMF to manifold learning problems. Experiments on a synthetic manifold learning dataset and three real datasets, including the MNIST handwritten dataset, a cryogenic electron microscopy dataset, and the Frey Face dataset, demonstrate the superiority of the proposed method over its competitors.

computer science, artificial intelligence,engineering, electrical & electronic
Accelerated Structured Matrix Factorization

Lorenzo Schiavon,Bernardo Nipoti,Antonio Canale

DOI: https://doi.org/10.1080/10618600.2023.2301072

2024-02-09

Journal of Computational and Graphical Statistics

Abstract:Matrix factorization exploits the idea that, in complex high-dimensional data, the actual signal typically lies in lower-dimensional structures. These lower dimensional objects provide useful insights, with interpretation favored by sparse structures. Sparsity, in addition, is beneficial in terms of regularization and, thus, to avoid over-fitting. By exploiting Bayesian shrinkage priors, we devise a computationally convenient approach for high-dimensional matrix factorization. The dependence between row and column entities is modeled by inducing flexible sparse patterns within factors. The availability of external information is accounted for in such a way that structures are allowed while not imposed. Inspired by boosting algorithms, we pair the proposed approach with a numerical strategy relying on a sequential inclusion and estimation of low-rank contributions, with a data-driven stopping rule. Practical advantages of the proposed approach are demonstrated by means of a simulation study and the analysis of soccer heatmaps obtained from new generation tracking data. Supplementary materials for this article are available online.

statistics & probability
Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Yan,Mingkui Tan,Ivor W. Tsang,Yi Yang,Qinfeng Shi,Chengqi Zhang

DOI: https://doi.org/10.1109/tkde.2018.2882197

IF: 9.235

2018-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Matrix factorization has been widely applied to various applications. With the fast development of storage and internet technologies, we have been witnessing a rapid increase of data. In this paper, we propose new algorithms for matrix factorization with the emphasis on efficiency. In addition, most existing methods of matrix factorization only consider a general smooth least square loss. Differently, many real-world applications have distinctive characteristics. As a result, different losses should be used accordingly. Therefore, it is beneficial to design new matrix factorization algorithms that are able to deal with both smooth and non-smooth losses. To this end, one needs to analyze the characteristics of target data and use the most appropriate loss based on the analysis. We particularly study two representative cases of low-rank matrix recovery, i.e., collaborative filtering for recommendation and high dynamic range imaging. To solve these two problems, we respectively propose a stage-wise matrix factorization algorithm by exploiting manifold optimization techniques. From our theoretical analysis, they are both are provably guaranteed to converge to a stationary point. Extensive experiments on recommender systems and high dynamic range imaging demonstrate the satisfactory performance and efficiency of our proposed method on large-scale real data.
Supplementary Material for “Median Matrix Completion: from Embarrassment to Optimality”

Weidong Liu,Xiaojun Mao,Raymond K. W. Wong

2020-01-01

Abstract:To prove the proposition, without loss of generality, we can assume that ‖A − A?‖F ≤ aN . It follows that f̂(0) = Equal contribution School of Mathematical Sciences and MoE Key Lab of Artificial Intelligence Shanghai Jiao Tong University, Shanghai, 200240, China School of Data Science, Fudan University, Shanghai, 200433, China Department of Statistics, Texas A&M University, College Station, TX 77843, U.S.A.. Correspondence to: Xiaojun Mao <maoxj@fudan.edu.cn>.
Stability of matrix factorization for collaborative filtering

Yu-Xiang Wang,Huan Xu

DOI: https://doi.org/10.48550/arXiv.1206.4640

2012-06-18

Numerical Analysis

Abstract:We study the stability vis a vis adversarial noise of matrix factorization algorithm for matrix completion. In particular, our results include: (I) we bound the gap between the solution matrix of the factorization method and the ground truth in terms of root mean square error; (II) we treat the matrix factorization as a subspace fitting problem and analyze the difference between the solution subspace and the ground truth; (III) we analyze the prediction error of individual users based on the subspace stability. We apply these results to the problem of collaborative filtering under manipulator attack, which leads to useful insights and guidelines for collaborative filtering system design.
Matrix Tri-Factorization With Manifold Regularizations For Zero-Shot Learning

Xing Xu,Fumin Shen,Yang Yang,Dongxiang Zhang,Heng Tao Shen,Jingkuan Song

DOI: https://doi.org/10.1109/CVPR.2017.217

2017-01-01

Abstract:Zero-shot learning (ZSL) aims to recognize objects of unseen classes with available training data from another set of seen classes. Existing solutions are focused on exploring knowledge transfer via an intermediate semantic embedding (e.g., attributes) shared between seen and unseen classes. In this paper, we propose a novel projection framework based on matrix tri-factorization with manifold regularizations. Specifically, we learn the semantic embedding projection by decomposing the visual feature matrix under the guidance of semantic embedding and class label matrices. By additionally introducing manifold regularizations on visual data and semantic embeddings, the learned projection can effectively capture the geometrical manifold structure residing in both visual and semantic spaces. To avoid the projection domain shift problem, we devise an effective prediction scheme by exploiting the test-time manifold structure. Extensive experiments on four benchmark datasets show that our approach significantly outperforms the state-of-the-arts, yielding an average improvement ratio by 7.4% and 31.9% for the recognition and retrieval task, respectively.
MatMat: Matrix Factorization by Matrix Fitting

Hao Wang

DOI: https://doi.org/10.1109/ICISCAE52414.2021.9590639

2021-12-06

Abstract:Matrix factorization is a widely adopted recommender system technique that fits scalar rating values by dot products of user feature vectors and item feature vectors. However, the formulation of matrix factorization as a scalar fitting problem is not friendly to side information incorporation or multi-task learning. In this paper, we replace the scalar values of the user rating matrix by matrices, and fit the matrix values by matrix products of user feature matrix and item feature matrix. Our framework is friendly to multitask learning and side information incorporation. We use popularity data as side information in our paper in particular to enhance the performance of matrix factorization techniques. In the experiment section, we prove the competence of our method compared with other approaches using both accuracy and fairness metrics. Our framework is an ideal substitute for tensor factorization in context-aware recommendation and many other scenarios.

Information Retrieval
Learning Matrix Factorization with Scalable Distance Metric and Regularizer

Shiping Wang,Yunhe Zhang,Xincan Lin,Lichao Su,Guobao Xiao,William Zhu,Yiqing Shi

DOI: https://doi.org/10.1016/j.neunet.2023.01.034

IF: 7.8

2023-01-01

Neural Networks

Abstract:Matrix factorization has always been an encouraging field, which attempts to extract discriminative features from high-dimensional data. However, it suffers from negative generalization ability and high computational complexity when handling large-scale data. In this paper, we propose a learnable deep matrix factorization via the projected gradient descent method, which learns multi-layer low-rank factors from scalable metric distances and flexible regularizers. Accordingly, solving a constrained matrix factorization problem is equivalently transformed into training a neural network with an appropriate activation function induced from the projection onto a feasible set. Distinct from other neural networks, the proposed method activates the connected weights not just the hidden layers. As a result, it is proved that the proposed method can learn several existing well-known matrix factorizations, including singular value decomposition, convex, nonnegative and semi-nonnegative matrix factorizations. Finally, comprehensive experiments demonstrate the superiority of the proposed method against other state-of-the-arts.
Stochastic Subsampling for Factorizing Huge Matrices

Arthur Mensch,Julien Mairal,Bertrand Thirion,Gael Varoquaux

DOI: https://doi.org/10.1109/TSP.2017.2752697

2017-10-30

Abstract:We present a matrix-factorization algorithm that scales to input matrices with both huge number of rows and columns. Learned factors may be sparse or dense and/or non-negative, which makes our algorithm suitable for dictionary learning, sparse component analysis, and non-negative matrix factorization. Our algorithm streams matrix columns while subsampling them to iteratively learn the matrix factors. At each iteration, the row dimension of a new sample is reduced by subsampling, resulting in lower time complexity compared to a simple streaming algorithm. Our method comes with convergence guarantees to reach a stationary point of the matrix-factorization problem. We demonstrate its efficiency on massive functional Magnetic Resonance Imaging data (2 TB), and on patches extracted from hyperspectral images (103 GB). For both problems, which involve different penalties on rows and columns, we obtain significant speed-ups compared to state-of-the-art algorithms.

Machine Learning,Optimization and Control,Neurons and Cognition
Simpler Proofs for Approximate Factor Models of Large Dimensions

Jushan Bai,Serena Ng

DOI: https://doi.org/10.48550/arXiv.2008.00254

2020-08-01

Abstract:Estimates of the approximate factor model are increasingly used in empirical work. Their theoretical properties, studied some twenty years ago, also laid the ground work for analysis on large dimensional panel data models with cross-section dependence. This paper presents simplified proofs for the estimates by using alternative rotation matrices, exploiting properties of low rank matrices, as well as the singular value decomposition of the data in addition to its covariance structure. These simplifications facilitate interpretation of results and provide a more friendly introduction to researchers new to the field. New results are provided to allow linear restrictions to be imposed on factor models.

Econometrics,Methodology
Verified Error Bounds for Matrix Decompositions

Siegfried M. Rump,Takeshi Ogita

DOI: https://doi.org/10.1137/24m165096x

IF: 1.908

2024-11-13

SIAM Journal on Matrix Analysis and Applications

Abstract:SIAM Journal on Matrix Analysis and Applications, Volume 45, Issue 4, Page 2155-2183, December 2024. In this note we consider common matrix factorizations such as LU decomposition of a square and rectangular matrix, Cholesky and QR decomposition, singular value decomposition for square and rectangular matrices, and eigen-, Schur, and Takagi decomposition. We first note that well-conditioned factors tend to be sensitive to perturbations of the input matrix, while ill-conditioned factors tend to be insensitive. It seems that this behavior has not been recognized in numerical analysis. We develop a formula for the relation between the condition number of the factor and its sensitivity with respect to input perturbations, and give reasons for that. Our main focus is to describe verification methods for the factors of the mentioned decompositions. That means proving existence of the factorization together with rigorous entrywise error bounds for the factors. Our goal is to develop algorithms requiring [math] operations for an [math] matrix with [math] and [math]. Moreover, bounds of high quality are aimed for, often not far from maximal accuracy. A main tool to achieve that is accurate dot products based on error-free transformations. Since preconditioning based on approximate inverses is used, our methods are restricted to full matrices.

mathematics, applied
Supplementary Material of "CUR Algorithm for Partially Observed Matrices"

Miao Xu,Rong Jin,Zhi‐Hua Zhou

2011-01-01

Abstract:∥M − PY (M)∥2 ≤ ∥Σ2∥2 + ∥Σ2Ψ2Ψ † 1∥2 where PY (M) projects column vectors in M in the subspace spanned by the column vectors in Y and † denotes the pseudoinverse. Theorem 2. (Derived From Theorem 2.2 of (Tropp, 2011)) Let X be a finite set of PSD matrices with dimension k (means the size of the square matrix is k× k). λmax(·) and λmin(·) calculate the maximum and minimum eigen value respectively.
Understanding the Sparsity: Augmented Matrix Factorization with Sampled Constraints on Unobservables.

Yongfeng Zhang,Min Zhang,Yi Zhang,Yiqun Liu,Shaoping Ma

DOI: https://doi.org/10.1145/2661829.2661976

2014-01-01

Abstract:An important problem of matrix completion/approximation based on Matrix Factorization (MF) algorithms is the existence of multiple global optima; this problem is especially serious when the matrix is sparse, which is common in real-world applications such as personalized recommender systems. In this work, we clarify data sparsity by bounding the solution space of MF algorithms. We present the conditions that an MF algorithm should satisfy for reliable completion of the unobservables, and we further propose to augment current MF algorithms with extra constraints constructed by compressive sampling on the unobserved values, which is well-motivated by the theoretical analysis. Model learning and optimal solution searching is conducted in a properly reduced solution space to achieve more accurate and efficient rating prediction performances. We implemented the proposed algorithms in the Map-Reduce framework, and comprehensive experimental results on Yelp and Dianping datasets verified the effectiveness and efficiency of the augmented matrix factorization algorithms.
Subspace-Constrained Quadratic Matrix Factorization: Algorithm and Applications

Zheng Zhai,Xiaohui Li

2024-11-07

Abstract:Matrix Factorization has emerged as a widely adopted framework for modeling data exhibiting low-rank structures. To address challenges in manifold learning, this paper presents a subspace-constrained quadratic matrix factorization model. The model is designed to jointly learn key low-dimensional structures, including the tangent space, the normal subspace, and the quadratic form that links the tangent space to a low-dimensional representation. We solve the proposed factorization model using an alternating minimization method, involving an in-depth investigation of nonlinear regression and projection subproblems. Theoretical properties of the quadratic projection problem and convergence characteristics of the alternating strategy are also investigated. To validate our approach, we conduct numerical experiments on synthetic and real-world datasets. Results demonstrate that our model outperforms existing methods, highlighting its robustness and efficacy in capturing core low-dimensional structures.

Machine Learning,Computer Vision and Pattern Recognition
Matrix Factor Analysis: from Least Squares to Iterative Projection

Yong He,Xinbing Kong,Long Yu,Xinsheng Zhang,Changwei Zhao

DOI: https://doi.org/10.1080/07350015.2023.2191676

2023-01-01

Journal of Business and Economic Statistics

Abstract:In this article, we study large-dimensional matrix factor models and estimate the factor loading matrices and factor score matrix by minimizing square loss function. Interestingly, the resultant estimators coincide with the Projected Estimators (PE) in Yu et al.(2022), which was proposed from the perspective of simultaneous reduction of the dimensionality and the magnitudes of the idiosyncratic error matrix. In other word, we provide a least-square interpretation of the PE for matrix factor model, which parallels to the least-square interpretation of the PCA for the vector factor model. We derive the convergence rates of the theoretical minimizers under sub-Gaussian tails. Considering the robustness to the heavy tails of the idiosyncratic errors, we extend the least squares to minimizing the Huber loss function, which leads to a weighted iterative projection approach to compute and learn the parameters. We also derive the convergence rates of the theoretical minimizers of the Huber loss function under bounded (2+e)th moment of the idiosyncratic errors. We conduct extensive numerical studies to investigate the empirical performance of the proposed Huber estimators relative to the state-of-the-art ones. The Huber estimators perform robustly and much better than existing ones when the data are heavy-tailed, and as a result can be used as a safe replacement in practice. An application to a Fama-French financial portfolio dataset demonstrates the empirical advantage of the Huber estimator.
Co-manifold Matrix Factorization.

Yunfeng Wu,Xuan Li,Zheng Qin

DOI: https://doi.org/10.1145/3436369.3436482

2020-01-01

Abstract:Matrix factorization plays a fundamental role in collaborative filtering. In collaborative filtering setting, the rating matrix R is very sparse. Thus, infinite number of matrices can fit the observed entries in the rating matrix. Without additional assumption, it is hard to decide which matrix can approximate the rating matrix well. The popular assumption is that the rating matrix is of low-rank. With this assumption, the rating matrix can be factorized into two latent factor matrices: P and Q. To improve the performance of the matrix factorization models, more assumptions can be made. A prevalent assumption is that rows of the latent factor matrix P or Q lie on a nonlinear manifold. Under this assumption, new matrix factorization models were proposed, which incorporate the geometric properties of the latent factor matrix P or Q into matrix factorization. In this paper, we assume instead that columns or rows of the rating matrix lie on a nonlinear manifold. With this assumption, we propose a new matrix factorization model, Co-Manifold Matrix Factorization (CoMMF), which incorporates the geometric properties of the rating matrix into matrix factorization. Experiments on real-world datasets show that our proposed model can outperform six state-of-the-art collaborative filtering methods.
Zipf Matrix Factorization : Matrix Factorization with Matthew Effect Reduction

Hao Wang

DOI: https://doi.org/10.48550/arXiv.2106.07347

2021-06-01

Information Retrieval

Abstract:Recommender system recommends interesting items to users based on users' past information history. Researchers have been paying attention to improvement of algorithmic performance such as MAE and precision@K. Major techniques such as matrix factorization and learning to rank are optimized based on such evaluation metrics. However, the intrinsic Matthew Effect problem poses great threat to the fairness of the recommender system, and the unfairness problem cannot be resolved by optimization of traditional metrics. In this paper, we propose a novel algorithm that incorporates Matthew Effect reduction with the matrix factorization framework. We demonstrate that our approach can boost the fairness of the algorithm and enhances performance evaluated by traditional metrics.
Matrix Factorizations via the Inverse Function Theorem

Paul W. Y. Lee,Paul W.Y. Lee

DOI: https://doi.org/10.48550/arXiv.1408.2611

2014-08-12

Classical Analysis and ODEs

Abstract:We give proofs of QR factorization, Cholesky's factorization, and LDU factorization using the inverse function theorem. As a consequence, we obtain analytic dependence of these matrix factorizations which does not follow immediately using Gaussian elimination.

“Self-Paced Learning for Matrix Factorization”: Supplementary Material

Self-Paced Learning for Matrix Factorization

Additional Online Supplement for “Inference for Approximate Factor Models: Random Missing and Cross Validation”

Quadratic Matrix Factorization With Applications to Manifold Learning

Accelerated Structured Matrix Factorization

Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Supplementary Material for “Median Matrix Completion: from Embarrassment to Optimality”

Stability of matrix factorization for collaborative filtering

Matrix Tri-Factorization With Manifold Regularizations For Zero-Shot Learning

MatMat: Matrix Factorization by Matrix Fitting

Learning Matrix Factorization with Scalable Distance Metric and Regularizer

Stochastic Subsampling for Factorizing Huge Matrices

Simpler Proofs for Approximate Factor Models of Large Dimensions

Verified Error Bounds for Matrix Decompositions

Supplementary Material of "CUR Algorithm for Partially Observed Matrices"

Understanding the Sparsity: Augmented Matrix Factorization with Sampled Constraints on Unobservables.

Subspace-Constrained Quadratic Matrix Factorization: Algorithm and Applications

Matrix Factor Analysis: from Least Squares to Iterative Projection

Co-manifold Matrix Factorization.

Zipf Matrix Factorization : Matrix Factorization with Matthew Effect Reduction

Matrix Factorizations via the Inverse Function Theorem