Abstract:This study considered an extension of a sparse regularization method with scaling, especially in thresholding methods that are simple and typical examples of sparse modeling. In this study, in the setting of a non-parametric orthogonal regression problem, we developed and analyzed a thresholding method in which soft thresholding estimators are independently expanded by empirical scaling values. The scaling values have a common hyper-parameter that is an order of expansion of an ideal scaling value to achieve hard thresholding. We simply refer to this estimator as a scaled soft thresholding estimator. The scaled soft thresholding method is a bridge method between soft and hard thresholding methods. This new estimator is indeed consistent with an adaptive LASSO estimator in the orthogonal case; i.e., it is thus an another derivation of an adaptive LASSO estimator. It is a general method that includes soft thresholding and non-negative garrote as special cases. We subsequently derived the degree of freedom of the scaled soft thresholding in calculating the Stein's unbiased risk estimate. We found that it is decomposed into the degree of freedom of soft thresholding and the remainder term connecting to the hard thresholding. As the degree of freedom reflects the degree of over-fitting, this implies that the scaled soft thresholding has an another source of over-fitting in addition to the number of un-removed components. The theoretical result was verified by a simple numerical example. In this process, we also focused on the non-monotonicity in the above remainder term of the degree of freedom and found that, in a sparse and large sample setting, it is mainly caused by useless components that are not related to the target function.

Accelerated Iterative Hard Thresholding Algorithm for $$l_0$$ Regularized Regression Problem

Adaptive Iterative Hard Thresholding for Least Absolute Deviation Problems with Sparsity Constraints

Smoothing fast iterative hard thresholding algorithm for $\ell_0$ regularized nonsmooth convex regression problem

Accelerate Randomized Coordinate Descent Iterative Hard Thresholding Methods for L(0) Regularized Convex Problems

A New Linear Convergence Result for the Iterative Soft Thresholding Algorithm

An accelerated proximal iterative hard thresholding method for $\ell_0$ minimization

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

Best Subset Selection for High-Dimensional Non-Smooth Models Using Iterative Hard Thresholding.

A New Proximal Iterative Hard Thresholding Method with Extrapolation for $$\ell _0$$ Minimization

Sparse Regularization: Convergence of Iterative Jumping Thresholding Algorithm

Efficient Stochastic Gradient Hard Thresholding

Convergence of Iterative Hard-Thresholding Algorithm with Continuation.

Sparse Convex Optimization via Adaptively Regularized Hard Thresholding

Hard Thresholding Pursuit with Continuation for ℓ0‐regularized Minimizations

ℓ 0 -Regularized High-Dimensional Accelerated Failure Time Model.

Optimal $k$-thresholding algorithms for sparse optimization problems

&Lt;formula Formulatype="inline"> <tex Notation="TeX">$L_{1/2}$</tex></formula> Regularization: A Thresholding Representation Theory and a Fast Solver

Loopless Semi-Stochastic Gradient Descent with Less Hard Thresholding for Sparse Learning

Dual Iterative Hard Thresholding: from Non-convex Sparse Minimization to Non-smooth Concave Maximization.

Bridging between Soft and Hard Thresholding by Scaling

Dual Iterative Hard Thresholding.