Abstract:Non-negative matrix factorization (NMF) approximates a non-negative matrix $X$ by a product of two non-negative low-rank factor matrices $W$ and $H$. NMF and its extensions minimize either the Kullback-Leibler divergence or the Euclidean distance between $X$ and $W^T H$ to model the Poisson noise or the Gaussian noise. In practice, when the noise distribution is heavy tailed, they cannot perform well. This paper presents Manhattan NMF (MahNMF) which minimizes the Manhattan distance between $X$ and $W^T H$ for modeling the heavy tailed Laplacian noise. Similar to sparse and low-rank matrix decompositions, MahNMF robustly estimates the low-rank part and the sparse part of a non-negative matrix and thus performs effectively when data are contaminated by outliers. We extend MahNMF for various practical applications by developing box-constrained MahNMF, manifold regularized MahNMF, group sparse MahNMF, elastic net inducing MahNMF, and symmetric MahNMF. The major contribution of this paper lies in two fast optimization algorithms for MahNMF and its extensions: the rank-one residual iteration (RRI) method and Nesterov's smoothing method. In particular, by approximating the residual matrix by the outer product of one row of W and one row of $H$ in MahNMF, we develop an RRI method to iteratively update each variable of $W$ and $H$ in a closed form solution. Although RRI is efficient for small scale MahNMF and some of its extensions, it is neither scalable to large scale matrices nor flexible enough to optimize all MahNMF extensions. Since the objective functions of MahNMF and its extensions are neither convex nor smooth, we apply Nesterov's smoothing method to recursively optimize one factor matrix with another matrix fixed. By setting the smoothing parameter inversely proportional to the iteration number, we improve the approximation accuracy iteratively for both MahNMF and its extensions.

On Naive Mean-Field Approximation for high-dimensional canonical GLMs

Sub-optimality of the Naive Mean Field approximation for proportional high-dimensional Linear Regression

Multivariate, Heteroscedastic Empirical Bayes via Nonparametric Maximum Likelihood

On Efficient and Scalable Computation of the Nonparametric Maximum Likelihood Estimator in Mixture Models

MahNMF: Manhattan Non-negative Matrix Factorization

Conjugate gradient methods for high-dimensional GLMMs

Optimal Gaussian approximations to the posterior for log-linear models with Diaconis-Ylvisaker priors

Posterior Concentration for Gaussian Process Priors under Rescaled and Hierarchical Matérn and Confluent Hypergeometric Covariance Functions

Manifold Optimization Assisted Gaussian Variational Approximation

High Dimensional Expectation-Maximization Algorithm: Statistical Optimization and Asymptotic Normality

Posterior Contraction rate and Asymptotic Bayes Optimality for one-group shrinkage priors in sparse normal means problem

Mean-field Variational Inference via Wasserstein Gradient Flow

On the Connection Between Non-negative Matrix Factorization and Latent Dirichlet Allocation

Robust Bayesian Nonnegative Matrix Factorization with Implicit Regularizers

A generalized mean field algorithm for variational inference in exponential families

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Bayesian mean-parameterized nonnegative binary matrix factorization

Are minimizers of the Onsager-Machlup functional strong posterior modes?

Adaptive Bayesian Regression on Data with Low Intrinsic Dimensionality

Partially factorized variational inference for high-dimensional mixed models

Estimating the mean and variance of a high-dimensional normal distribution using a mixture prior