Abstract:This study considered an extension of a sparse regularization method with scaling, especially in thresholding methods that are simple and typical examples of sparse modeling. In this study, in the setting of a non-parametric orthogonal regression problem, we developed and analyzed a thresholding method in which soft thresholding estimators are independently expanded by empirical scaling values. The scaling values have a common hyper-parameter that is an order of expansion of an ideal scaling value to achieve hard thresholding. We simply refer to this estimator as a scaled soft thresholding estimator. The scaled soft thresholding method is a bridge method between soft and hard thresholding methods. This new estimator is indeed consistent with an adaptive LASSO estimator in the orthogonal case; i.e., it is thus an another derivation of an adaptive LASSO estimator. It is a general method that includes soft thresholding and non-negative garrote as special cases. We subsequently derived the degree of freedom of the scaled soft thresholding in calculating the Stein's unbiased risk estimate. We found that it is decomposed into the degree of freedom of soft thresholding and the remainder term connecting to the hard thresholding. As the degree of freedom reflects the degree of over-fitting, this implies that the scaled soft thresholding has an another source of over-fitting in addition to the number of un-removed components. The theoretical result was verified by a simple numerical example. In this process, we also focused on the non-monotonicity in the above remainder term of the degree of freedom and found that, in a sparse and large sample setting, it is mainly caused by useless components that are not related to the target function.

Bridging between Soft and Hard Thresholding by Scaling

Adaptive scaling for soft-thresholding estimator

Adaptive Iterative Hard Thresholding for Least Absolute Deviation Problems with Sparsity Constraints

On an improvement of LASSO by scaling

Adaptive Lasso in high-dimensional settings

Scaled minimax optimality in high-dimensional linear regression: A non-convex algorithmic regularization approach

A New Linear Convergence Result for the Iterative Soft Thresholding Algorithm

Smoothing fast iterative hard thresholding algorithm for $\ell_0$ regularized nonsmooth convex regression problem

Sparse Convex Optimization via Adaptively Regularized Hard Thresholding

Optimal $k$-thresholding algorithms for sparse optimization problems

Efficient Stochastic Gradient Hard Thresholding

CANONICAL THRESHOLDING FOR NON-SPARSE HIGH-DIMENSIONAL LINEAR REGRESSION

Scaled sparse linear regression

Probabilistic Iterative Hard Thresholding for Sparse Learning

Stability and Risk Bounds of Iterative Hard Thresholding

Characterizing the SLOPE Trade-off: A Variational Perspective and the Donoho-Tanner Limit

Separable Bregman Framework for Sparsity Constrained Nonlinear Optimization

Randomized Iterative Hard Thresholding for Sparse Approximations

Loopless Semi-Stochastic Gradient Descent with Less Hard Thresholding for Sparse Learning

A minimax optimal approach to high-dimensional double sparse linear regression

Scaled Proximal Gradient Methods for Sparse Optimization Problems