Abstract:The Iterative Hard Thresholding (IHT) algorithm has been considered extensively as an effective deterministic algorithm for solving sparse optimizations. The IHT algorithm benefits from the information of the batch (full) gradient at each point and this information is a crucial key for the convergence analysis of the generated sequence. However, this strength becomes a weakness when it comes to machine learning and high dimensional statistical applications because calculating the batch gradient at each iteration is computationally expensive or impractical. Fortunately, in these applications the objective function has a summation structure that can be taken advantage of to approximate the batch gradient by the stochastic mini-batch gradient. In this paper, we study the mini-batch Stochastic IHT (SIHT) algorithm for solving the sparse optimizations. As opposed to previous works where increasing and variable mini-batch size is necessary for derivation, we fix the mini-batch size according to a lower bound that we derive and show our work. To prove stochastic convergence of the objective value function we first establish a critical sparse stochastic gradient descent property. Using this stochastic gradient descent property we show that the sequence generated by the stochastic mini-batch SIHT is a supermartingale sequence and converges with probability one. Unlike previous work we do not assume the function to be a restricted strongly convex. To the best of our knowledge, in the regime of sparse optimization, this is the first time in the literature that it is shown that the sequence of the stochastic function values converges with probability one by fixing the mini-batch size for all steps.

Dual Iterative Hard Thresholding.

Dual Iterative Hard Thresholding: from Non-convex Sparse Minimization to Non-smooth Concave Maximization.

Adaptive Iterative Hard Thresholding for Least Absolute Deviation Problems with Sparsity Constraints

Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization.

Best Subset Selection for High-Dimensional Non-Smooth Models Using Iterative Hard Thresholding.

Efficient Stochastic Gradient Hard Thresholding

Stability and Risk Bounds of Iterative Hard Thresholding

Learning Sparse Distributions using Iterative Hard Thresholding

Accelerated Iterative Hard Thresholding Algorithm for $$l_0$$ Regularized Regression Problem

Probabilistic Iterative Hard Thresholding for Sparse Learning

Accelerate Randomized Coordinate Descent Iterative Hard Thresholding Methods for L(0) Regularized Convex Problems

Iterative Hard Thresholding for Weighted Sparse Approximation

Exact Recovery of Hard Thresholding Pursuit.

Sparse Convex Optimization via Adaptively Regularized Hard Thresholding

Optimal $k$-thresholding algorithms for sparse optimization problems

Convergence of the mini-batch SIHT algorithm

Smoothing fast iterative hard thresholding algorithm for $\ell_0$ regularized nonsmooth convex regression problem

A minimax optimal approach to high-dimensional double sparse linear regression

Randomized Iterative Hard Thresholding for Sparse Approximations

Loopless Semi-Stochastic Gradient Descent with Less Hard Thresholding for Sparse Learning

Distributed Iterative Hard Thresholding for Variable Selection in Tobit Models