Abstract:Solving linear inverse problems plays a crucial role in numerous applications. Algorithm unfolding based, model-aware data-driven approaches have gained significant attention for effectively addressing these problems. Learned iterative soft-thresholding algorithm (LISTA) and alternating direction method of multipliers compressive sensing network (ADMM-CSNet) are two widely used such approaches, based on ISTA and ADMM algorithms, respectively. In this work, we study optimization guarantees, i.e., achieving near-zero training loss with the increase in the number of learning epochs, for finite-layer unfolded networks such as LISTA and ADMM-CSNet with smooth soft-thresholding in an over-parameterized (OP) regime. We achieve this by leveraging a modified version of the Polyak-Lojasiewicz, denoted PL$^*$, condition. Satisfying the PL$^*$ condition within a specific region of the loss landscape ensures the existence of a global minimum and exponential convergence from initialization using gradient descent based methods. Hence, we provide conditions, in terms of the network width and the number of training samples, on these unfolded networks for the PL$^*$ condition to hold. We achieve this by deriving the Hessian spectral norm of these networks. Additionally, we show that the threshold on the number of training samples increases with the increase in the network width. Furthermore, we compare the threshold on training samples of unfolded networks with that of a standard fully-connected feed-forward network (FFNN) with smooth soft-thresholding non-linearity. We prove that unfolded networks have a higher threshold value than FFNN. Consequently, one can expect a better expected error for unfolded networks than FFNN.

Learned ISTA with Error-based Thresholding for Adaptive Sparse Coding

Learned Extragradient ISTA with Interpretable Residual Structures for Sparse Coding.

Sparse Coding with Gated Learned ISTA

Adaptive Iterative Hard Thresholding for Least Absolute Deviation Problems with Sparsity Constraints

Compressed Lista Exploiting Toeplitz Structure

A New Linear Convergence Result for the Iterative Soft Thresholding Algorithm

Adaptive Iterated Shrinkage Thresholding-Based Lp-Norm Sparse Representation for Hyperspectral Imagery Target Detection

Sparse solution of underdetermined linear equations via adaptively iterative thresholding

Adaptive Thresholding for Sparse Covariance Matrix Estimation

Deep Sparse Coding Using Optimized Linear Expansion of Thresholds

Learn the Approximation Distribution of Sparse Coding with Mixture Sparsity Network.

Randomized Iterative Hard Thresholding for Sparse Approximations

Trainable ISTA for Sparse Signal Recovery

Learning Fast Approximations of Sparse Nonlinear Regression

Learning Sparse Distributions using Iterative Hard Thresholding

Variational Learning ISTA

Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding

A Tight Bound of Modified Iterative Hard Thresholding Algorithm for Compressed Sensing

A Generalized Iterated Shrinkage Algorithm for Non-convex Sparse Coding

A Modified Iterative Hard Thresholding Algorithm

Linear convergence of adaptively iterative thresholding algorithms for compressed sensing