Abstract:Convex regularizers is popular for sparse and low-rank learning, mainly due to their nice statistical and optimization guarantee. However, they often lead to bias estimation, thus the sparsity and accuracy is not as good as desired. This motivates replacement of convex regularizers with nonconvex one, and recently many nonconvex regularizers have been proposed and indeed better performance than convex ones can be achieved. In this survey, we summary and compare algorithms which can be applied on learning with nonconvex regularizers. 1 General Non-Convex Optimization Consider general non-convex optimization problem 1 min x g(x) (1) Assumption 1 (Difference of Convex [1, 18, 14]). g(x) can be decomposed as difference of two convex (DoC) functions, i.e. g(x) = ĝ(x)− ḡ(x). We are more interested in non-smooth g(x), otherwise, (1) can be solve simply with gradient descent. With Assumption 1, (1) can be expressed as min x {g(x) ≡ ĝ(x)− ḡ(x)} where ĝ(x) and ḡ(x) are two convex functions. Definition 1 (Critical point [1]). Under Assumption 1, a point x∗ is a critical point of g(x), if it satisfies 0 ∈ ∂ĝ(x∗)− ∂ḡ(x∗). Remark. For smooth functions, critical point is defined as point where its gradient is zero. Difference of Convex programming (DCP) [1] is a general technique to optimize (1). Basically, it uses first order expansion to approximate ḡ(x), and generates {xt, yt} by xt+1 = arg min x ĝ(x)− 〈x− xt, yt〉 , yt ∈ ∂ḡ(xt) (2) Though g(x) is not convex, we can see the sub-problem in (2) becomes convex. Therefore, algorithms for convex optimization can be applied. However, the sub-problem may not be easy to solve, and this makes DCP slow in real world applications. This method is used in [4, 19] for sparse coding with non-convex regularizers. Theorem 1.1 (Convergence of DCP). Let Assumption 1 hold, then xt generated by (2) will converge to a critical point of (1). All functions mentioned in this paper are assumed bounded from below

A Comparison of First-order Algorithms for Machine Learning

Accelerated First-Order Optimization Algorithms for Machine Learning.

Comparisons Are All You Need for Optimizing Smooth Functions

First-order and Stochastic Optimization Methods for Machine Learning

A Comparison of Optimization Algorithms for Deep Learning

Optimization with First Order Algorithms

Comparison of Proximal First-Order Primal and Primal-Dual algorithms via Performance Estimation

A survey of deep learning optimizers -- first and second order methods

An Aggressive Reduction on the Complexity of Optimization for Non-Strongly Convex Objectives

A Survey of Optimization Methods From a Machine Learning Perspective

A Comparison of Algorithms for Learning with Nonconvex Regularization

Comparison of SVM Optimization Techniques in the Primal

Review Non-convex Optimization Method for Machine Learning

Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

On quasi-convex smooth optimization problems by a comparison oracle

Accelerated First-Order Optimization under Nonlinear Constraints

Formalization of Complexity Analysis of the First-order Algorithms for Convex Optimization

Machine Learning Problems from Optimization Perspective

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

Accelerated Optimization for Machine Learning