Abstract:Convex regularizers is popular for sparse and low-rank learning, mainly due to their nice statistical and optimization guarantee. However, they often lead to bias estimation, thus the sparsity and accuracy is not as good as desired. This motivates replacement of convex regularizers with nonconvex one, and recently many nonconvex regularizers have been proposed and indeed better performance than convex ones can be achieved. In this survey, we summary and compare algorithms which can be applied on learning with nonconvex regularizers. 1 General Non-Convex Optimization Consider general non-convex optimization problem 1 min x g(x) (1) Assumption 1 (Difference of Convex [1, 18, 14]). g(x) can be decomposed as difference of two convex (DoC) functions, i.e. g(x) = ĝ(x)− ḡ(x). We are more interested in non-smooth g(x), otherwise, (1) can be solve simply with gradient descent. With Assumption 1, (1) can be expressed as min x {g(x) ≡ ĝ(x)− ḡ(x)} where ĝ(x) and ḡ(x) are two convex functions. Definition 1 (Critical point [1]). Under Assumption 1, a point x∗ is a critical point of g(x), if it satisfies 0 ∈ ∂ĝ(x∗)− ∂ḡ(x∗). Remark. For smooth functions, critical point is defined as point where its gradient is zero. Difference of Convex programming (DCP) [1] is a general technique to optimize (1). Basically, it uses first order expansion to approximate ḡ(x), and generates {xt, yt} by xt+1 = arg min x ĝ(x)− 〈x− xt, yt〉 , yt ∈ ∂ḡ(xt) (2) Though g(x) is not convex, we can see the sub-problem in (2) becomes convex. Therefore, algorithms for convex optimization can be applied. However, the sub-problem may not be easy to solve, and this makes DCP slow in real world applications. This method is used in [4, 19] for sparse coding with non-convex regularizers. Theorem 1.1 (Convergence of DCP). Let Assumption 1 hold, then xt generated by (2) will converge to a critical point of (1). All functions mentioned in this paper are assumed bounded from below

Learning Nonlinear Functions Using Regularized Greedy Forest

Efficient non-greedy optimization of decision trees

Local Linear Forests

Riemann-Lebesgue Forest for Regression

Score Function Gradient Estimation to Widen the Applicability of Decision-Focused Learning

Learning Decision Trees and Forests with Algorithmic Recourse

Exploiting random projections and sparsity with random forests and gradient boosting methods -- Application to multi-label and multi-output learning, random forest model compression and leveraging input sparsity

Learning Linear Programs from Optimal Decisions

Nonlinear Optimal Classification Trees

Minimax Adaptive Boosting for Online Nonparametric Regression

Nonlinear Collaborative Scheme for Deep Neural Networks.

Non-asymptotic Properties of Generalized Mondrian Forests in Statistical Learning

Jointly learning relevant subgraph patterns and nonlinear models of their indicators

Improving Deep Forest by Exploiting High-order Interactions

Amortized Active Learning for Nonparametric Functions

Mixed-Integer Convex Nonlinear Optimization with Gradient-Boosted Trees Embedded

A Comparison of Algorithms for Learning with Nonconvex Regularization

Greedy function approximation: A gradient boosting machine.

Learning Nonparametric Forest Graphical Models With Prior Information

Nonstationary Nonparametric Online Learning: Balancing Dynamic Regret and Model Parsimony

Statistical-Computational Trade-offs for Greedy Recursive Partitioning Estimators