Iteratively re-weighted least squares minimization for sparse recovery

Ingrid Daubechies,Ronald DeVore,Massimo Fornasier,C. Sinan Gunturk

DOI: https://doi.org/10.48550/arXiv.0807.0575

2008-07-03

Abstract:We analyze an Iteratively Re-weighted Least Squares (IRLS) algorithm for promoting l1-minimization in sparse and compressible vector recovery. We prove its convergence and we estimate its local rate. We show how the algorithm can be modified in order to promote lt-minimization for t<1, and how this modification produces superlinear rates of convergence.

Numerical Analysis

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to recover the sparse vector \(x\in\mathbb{R}^N\) from the under - determined linear system \(\Phi x = y\) under an \(m\times N\) matrix \(\Phi\) (where \(m < N\)) that satisfies a certain condition (called the restricted isometry property or RIP). Specifically:

1. **Problem Background**:
- The under - determined linear system \(\Phi x = y\) usually has infinitely many solutions.
- If the solution \(x\) is sparse (that is, most elements are zero), then the unique sparse solution can be found by minimizing the \(\ell_1\)-norm.

2. **Existing Methods**:
- By solving the linear programming problem, the solution \(x\) with the minimum \(\ell_1\)-norm can be found:
\[
x=\arg\min_{z\in\Phi^{-1}(y)}\|z\|_{\ell_1}
\]
- This method is numerically feasible but may not be the most efficient.

3. **The Method Proposed in the Paper**:
- The paper studies an alternative method: the iterative re - weighted least squares method (IRLS). This method gradually approximates the sparse solution by iteratively updating the weights \(w\).
- In each iteration, the algorithm finds the solution \(x^{(n + 1)}\) with the minimum \(\ell_2(w)\)-norm, where the weight \(w^{(n)}_i=\left[|x^{(n)}_i|^2+\epsilon_n^2\right]^{-1/2}\).
- As the iteration progresses, the weights are gradually adjusted so that the algorithm can converge to the sparse solution.

4. **Main Contributions**:
- The paper proves that when the matrix \(\Phi\) satisfies the RIP condition, the IRLS algorithm will converge for any \(y\).
- If \(\Phi^{-1}(y)\) contains a sparse solution, then the algorithm will converge to this sparse solution and converge exponentially fast as it approaches the limit.
- The paper also analyzes the case of using heavier weights \(w^{(n)}_i=\left[|x^{(n)}_i|^2+\epsilon_n^2\right]^{-1+\tau/2}\) and proves that its local convergence speed is faster, approaching quadratic convergence.

In summary, this paper aims to provide an efficient and reliable iterative method for recovering sparse vectors, especially having important applications in fields such as compressed sensing.