Abstract:Block majorization-minimization (BMM) is a simple iterative algorithm for nonconvex constrained optimization that sequentially minimizes majorizing surrogates of the objective function in each block coordinate while the other coordinates are held fixed. BMM entails a large class of optimization algorithms such as block coordinate descent and its proximal-point variant, expectation-minimization, and block projected gradient descent. We establish that for general constrained nonconvex optimization, BMM with strongly convex surrogates can produce an $\epsilon$-stationary point within $O(\epsilon^{-2}(\log \epsilon^{-1})^{2})$ iterations and asymptotically converges to the set of stationary points. Furthermore, we propose a trust-region variant of BMM that can handle surrogates that are only convex and still obtain the same iteration complexity and asymptotic stationarity. These results hold robustly even when the convex sub-problems are inexactly solved as long as the optimality gaps are summable. As an application, we show that a regularized version of the celebrated multiplicative update algorithm for nonnegative matrix factorization by Lee and Seung has iteration complexity of $O(\epsilon^{-2}(\log \epsilon^{-1})^{2})$. The same result holds for a wide class of regularized nonnegative tensor decomposition algorithms as well as the classical block projected gradient descent algorithm. These theoretical results are validated through various numerical experiments.

The appeals of quadratic majorization–minimization

Relaxed Majorization-Minimization for Non-Smooth and Non-Convex Optimization

Nonconvex Optimization via MM Algorithms: Convergence Theory

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

Block majorization-minimization with diminishing radius for constrained nonconvex optimization

Distance Majorization and Its Applications

Universal Majorization-Minimization Algorithms

Stochastic Variance-Reduced Majorization-Minimization Algorithms

Generalized Majorization-Minimization for Non-Convex Optimization.

Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning

Algorithmic Design of Majorizers for Large-Scale Inverse Problems

On the Global Convergence of Majorization Minimization Algorithms for Nonconvex Optimization Problems.

Convergence Analysis of Generalized ADMM with Majorization for Linearly Constrained Composite Convex Optimization

Convergence and complexity of block majorization-minimization for constrained block-Riemannian optimization

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

Majorizing Functions and Convergence of the Gauss-Newton Method for Convex Composite Optimization

Convergence analysis of stochastic higher-order majorization-minimization algorithms

Chapter 9: MM Algorithms

On the convergence of Block Majorization-Minimization algorithms on the Grassmann Manifold

Proximal Dogleg Opportunistic Majorization for Nonconvex and Nonsmooth Optimization

Min-Max Framework for Majorization-Minimization Algorithms in Signal Processing Applications: An Overview