Primal and dual alternating direction algorithms for l1-l1-norm minimization problems in compressive sensing

XiaoYunhai,ZhuHong,WuSoon-Yi

2013-01-01

Computational Optimization and Applications

Abstract:In this paper, we propose, analyze and test primal and dual versions of the alternating direction algorithm for the sparse signal reconstruction from its major noise contained observation data. The...

What problem does this paper attempt to address?

Alternating Direction Algorithms for $\Ell_1$-Problems in Compressive Sensing

Junfeng Yang,Yin Zhang

DOI: https://doi.org/10.1137/090777761

IF: 2.968

2009-01-01

SIAM Journal on Scientific Computing

Abstract:In this paper, we propose and study the use of alternating direction algorithms for several $\ell_1$-norm minimization problems arising from sparse solution recovery in compressive sensing, including the basis pursuit problem, the basis pursuit denoising problems of both unconstrained and constrained forms, and others. We present and investigate two classes of algorithms derived from either the primal or the dual form of $\ell_1$-problems. The construction of the algorithms consists of two main steps: (1) to reformulate an $\ell_1$-problem into one having blockwise separable objective functions by adding new variables and constraints; and (2) to apply an exact or inexact alternating direction method to the augmented Lagrangian function of the resulting problem. The derived alternating direction algorithms can be regarded as first-order primal-dual algorithms because both primal and dual variables are updated at every iteration. Convergence properties of these algorithms are established or restated when they already exist. Extensive numerical experiments are performed, using randomized partial Walsh-Hadamard sensing matrices, to demonstrate the versatility and effectiveness of the proposed approach. Moreover, we present numerical results to emphasize two practically important but perhaps overlooked points: (i) that algorithm speed should be evaluated relative to appropriate solution accuracy; and (ii) that when erroneous measurements possibly exist, the $\ell_1$-fidelity should generally be preferable to the $\ell_2$-fidelity.
Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Hang Xu,Song Li,Junhong Lin

DOI: https://doi.org/10.1007/s10444-024-10163-2

2024-01-01

Advances in Computational Mathematics

Abstract:Many problems in data science can be treated as recovering structural signals from a set of linear measurements, sometimes perturbed by dense noise or sparse corruptions. In this paper, we develop a unified framework of considering a nonsmooth formulation with sparse or low-rank constraint for meeting the challenges of mixed noises—bounded noise and sparse noise. We show that the nonsmooth formulations of the problems can be well solved by the projected subgradient methods at a rapid rate when initialized at any points. Consequently, nonsmooth loss functions ( ℓ _1 -minimization programs) are naturally robust against sparse noise. Our framework simplifies and generalizes the existing analyses including compressed sensing, matrix sensing, quadratic sensing, and bilinear sensing. Motivated by recent work on the stochastic gradient method, we also give some experimentally and theoretically preliminary results about the projected stochastic subgradient method.
On the Null Space Property of L Q-Minimization for 0 < Q ≤ 1 in Compressed Sensing

Yi Gao,Jigen Peng,Shigang Yue,Yuan Zhao

DOI: https://doi.org/10.1155/2015/579853

2015-01-01

Journal of Function Spaces

Abstract:The paper discusses the relationship between the null space property (NSP) and the l q -minimization in compressed sensing. Several versions of the null space property, that is, the l q stable NSP, the l q robust NSP, and the l q,p robust NSP for 0 < p ≤ q < 1 based on the standard l q NSP, are proposed, and their equivalent forms are derived. Consequently, reconstruction results for the l q -minimization can be derived easily under the NSP condition and its equivalent form. Finally, the l q NSP is extended to the l q -synthesis modeling and the mixed l 2 /l q -minimization, which deals with the dictionary-based sparse signals and the block sparse signals, respectively.
A Primal Dual Active Set Algorithm with Continuation for Compressed Sensing

Qibin Fan,Yuling Jiao,Xiliang Lu

DOI: https://doi.org/10.1109/TSP.2014.2362880

2013-12-26

Abstract:The success of compressed sensing relies essentially on the ability to efficiently find an approximately sparse solution to an under-determined linear system. In this paper, we developed an efficient algorithm for the sparsity promoting $\ell_1$-regularized least squares problem by coupling the primal dual active set strategy with a continuation technique (on the regularization parameter). In the active set strategy, we first determine the active set from primal and dual variables, and then update the primal and dual variables by solving a low-dimensional least square problem on the active set, which makes the algorithm very efficient. The continuation technique globalizes the convergence of the algorithm, with provable global convergence under restricted isometry property (RIP). Further, we adopt two alternative methods, i.e., a modified discrepancy principle and a Bayesian information criterion, to choose the regularization parameter. Numerical experiments indicate that our algorithm is very competitive with state-of-the-art algorithms in terms of accuracy and efficiency.

Information Theory,Optimization and Control
Alternating direction algorithms for $\ell_0$ regularization in compressed sensing.

Chao-Bing Song,Shu-Tao Xia

2016-01-01

Abstract:In this paper we propose three iterative greedy algorithms for compressed sensing, called emph{iterative alternating direction} (IAD), emph{normalized iterative alternating direction} (NIAD) and emph{alternating direction pursuit} (ADP), which stem from the iteration steps of alternating direction method of multiplier (ADMM) for $ell_0$-regularized least squares ($ell_0$-LS) and can be considered as the alternating direction versions of the well-known iterative hard thresholding (IHT), normalized iterative hard thresholding (NIHT) and hard thresholding pursuit (HTP) respectively. Firstly, relative to the general iteration steps of ADMM, the proposed algorithms have no splitting or dual variables in iterations and thus the dependence of the current approximation on past iterations is direct. Secondly, provable theoretical guarantees are provided in terms of restricted isometry property, which is the first theoretical guarantee of ADMM for $ell_0$-LS to the best of our knowledge. Finally, they outperform the corresponding IHT, NIHT and HTP greatly when reconstructing both constant amplitude signals with random signs (CARS signals) and Gaussian signals.
A Primal Dual Active Set with Continuation Algorithm for the ℓ^0-Regularized Optimization Problem

Yuling Jiao,Bangti Jin,Xiliang Lu

DOI: https://doi.org/10.1016/j.acha.2014.10.001

IF: 2.974

2014-01-01

Applied and Computational Harmonic Analysis

Abstract:We develop a primal dual active set with continuation algorithm for solving the ℓ0-regularized least-squares problem that frequently arises in compressed sensing. The algorithm couples the primal dual active set method with a continuation strategy on the regularization parameter. At each inner iteration, it first identifies the active set from both primal and dual variables, and then updates the primal variable by solving a (typically small) least-squares problem defined on the active set, from which the dual variable can be updated explicitly. Under certain conditions on the sensing matrix, i.e., mutual incoherence property or restricted isometry property, and the noise level, a finite step global convergence of the overall algorithm is established. Extensive numerical examples are presented to illustrate the efficiency and accuracy of the algorithm and its convergence behavior.
A Primal Douglas–Rachford Splitting Method for the Constrained Minimization Problem in Compressive Sensing

Yongchao Yu,Jigen Peng,Xuanli Han,Angang Cui

DOI: https://doi.org/10.1007/s00034-017-0498-5

2017-01-01

Abstract:Compressive sensing has achieved great success in many scientific research fields. It has revealed that sparse signals can be stably recovered from a small number of noisy measurements by solving the constrained convex ${\ell }_1$-minimization problem. In practice, a faster algorithm for solving this optimization problem is the key to compressive sensing. The Douglas–Rachford splitting method is a well-known operator splitting method that has been widely applied for solving a certain class of convex composite problems. In particular, its dual application results in the popular alternating direction method of multipliers (ADMM). In this paper, we reformulate the constrained convex ${\ell }_1$-minimization problem as a convex composite problem with a special structure and then apply the primal Douglas–Rachford splitting method to solve it. The computational cost of the developed algorithm in each iteration is dominated by the projection onto the constraint set. A fast and efficient method of computing the projection is proposed. Numerical results show that the developed algorithm performs better than the popular NESTA and LADMM (inexact ADMM) in terms of accuracy and run time for large-scale sparse signal recovery.
A splitting primal-dual proximity algorithm for solving composite optimization problems

Yu-Chao Tang,Chuan-Xi Zhu,Meng Wen,Ji-Gen Peng

DOI: https://doi.org/10.1007/s10114-016-5625-x

2015-08-23

Abstract:Our work considers the optimization of the sum of a non-smooth convex function and a finite family of composite convex functions, each one of which is composed of a convex function and a bounded linear operator. This type of problem is associated with many interesting challenges encountered in the image restoration and image reconstruction fields. We developed a splitting primal-dual proximity algorithm to solve this problem. Further, we propose a preconditioned method, of which the iterative parameters are obtained without the need to know some particular operator norm in advance. Theoretical convergence theorems are presented. We then apply the proposed methods to solve a total variation regularization model, in which the L2 data error function is added to the L1 data error function. The main advantageous feature of this model is its capability to combine different loss functions. The numerical results obtained for computed tomography (CT) image reconstruction demonstrated the ability of the proposed algorithm to reconstruct an image with few and sparse projection views while maintaining the image quality.

Optimization and Control
Study on L_1 over L_2 Minimization for Nonnegative Signal Recovery

Min Tao,Xiao-Ping Zhang

DOI: https://doi.org/10.1007/s10915-023-02225-2

2023-01-01

Journal of Scientific Computing

Abstract:In this paper, we carry out a comprehensive study for the unconstrained L_1 over L_2 sparsity promoting model, widely used in the regime of coherent dictionaries for recovering nonnegative sparse signals. First, we prove the existence of global solutions. Second, we analyze the sparse property of any local minimizer of the L_1/L_2 model. This property serves as a certificate to rule out the nonlocal minimizers. Third, we focus on algorithmic development on the unconstrained model with nonnegative constraint. We derive an analytical solution for the proximal operator of the L_1 / L_2 with nonnegative constraint. Then, we apply the alternating direction method of multipliers in a particular splitting way, referred to as ADMM _p^+ . We establish its global convergence to a d-stationary solution (sharpest stationary) by verifying the Lyapunov function with the Kurdyka-Łojasiewicz property instead of imposing. Extensive numerical simulations confirm the superiority of ADMM _p^+ over the existing state-of-the-art methods in nonnegative sparse recovery.
Analysis of the Ratio of ℓ1 and ℓ2 Norms in Compressed Sensing

Yiming Xu,Akil Narayan,Hoang Tran,Clayton G. Webster

DOI: https://doi.org/10.1016/j.acha.2021.06.006

IF: 2.974

2021-01-01

Applied and Computational Harmonic Analysis

Abstract:We study the ratio of l(1) and l(2) norms (l(1)/l(2)) as a sparsity-promoting objective in compressed sensing. We first propose a novel criterion that guarantees that an s-sparse signal is the local minimizer of the l(1)/l(2) objective; our criterion is interpretable and useful in practice. We also give the first uniform recovery condition using a geometric characterization of the null space of the measurement matrix, and show that this condition is satisfied for a class of random matrices. We also present analysis on the robustness of the procedure when noise pollutes data. Numerical experiments are provided that compare l(1)/l(2) with some other popular non-convex methods in compressed sensing. Finally, we propose a novel initialization approach to accelerate the numerical optimization procedure. We call this initialization approach support selection, and we demonstrate that it empirically improves the performance of existing l(1)/l(2) algorithms. (C) 2021 Elsevier Inc. All rights reserved.
The variant of the iterative shrinkage-thresholding algorithm for minimization of the l 1 over l ∞ norms

Jun Wang,Qiang Ma

DOI: https://doi.org/10.1016/j.sigpro.2023.109104

IF: 4.729

2023-05-22

Signal Processing

Abstract:In this paper, we study minimization of the ratio of l1 and l∞ norms ( l1/l∞ ) as a nonconvex and sparsity-promoting metric for solving the unconstrained compressed sensing problems. To design some efficient algorithms, we derive a closed-form solution to the proximal operator of the l1/l∞ function. With the newly variant of the shrinkage operator, we propose a novel variant of fast iterative shrinkage-thresholding algorithm (FISTA) and a specific variable-splitting scheme of the alternating direction method of multipliers (ADMM) which is guaranteed to have sub-sequential convergence. Experimentally, we construct extensive numerical simulations to demonstrate the efficiency of these two proposed approaches over the state-of-the-art algorithms in sparse recovery.

engineering, electrical & electronic
Unified Analysis on L1 over L2 Minimization for Signal Recovery

Min Tao,Xiao-Ping Zhang

DOI: https://doi.org/10.36227/techrxiv.15086277

2023-01-01

Abstract:In this paper, we carry out a unified study for L_1 over L_2 sparsity promoting models, which are widely used in the regime of coherent dictionaries for recovering sparse nonnegative/arbitrary signal. First, we provide the exact recovery condition on both the constrained and the unconstrained models for a broad set of signals. Next, we prove the solution existence of these L_{1}/L_{2} models under the assumption that the null space of the measurement matrix satisfies the $s$-spherical section property. Then by deriving an analytical solution for the proximal operator of the L_{1} / L_{2} with nonnegative constraint, we develop a new alternating direction method of multipliers based method (ADMM$_p^+$) to solve the unconstrained model. We establish its global convergence to a d-stationary solution (sharpest stationary) and its local linear convergence under certain conditions. Numerical simulations on two specific applications confirm the superior of ADMM$_p^+$ over the state-of-the-art methods in sparse recovery. ADMM$_p^+$ reduces computational time by about $95\%\sim99\%$ while achieving a much higher accuracy compared to commonly used scaled gradient projection method for wavelength misalignment problem.
A Block Alternating Optimization Method for Direction-of-Arrival Estimation with Nested Array.

Yunmei Shi,Xing-Peng Mao,Chunlei Zhao,Yong-Tan Liu

DOI: https://doi.org/10.1109/ACCESS.2019.2921967

IF: 3.9

2019-01-01

IEEE Access

Abstract:In this paper, direction-of-arrival estimation using nested array is studied in the framework of sparse signal representation. With the vectorization operator, a new real-valued non-negative sparse signal recovery model, which has a wider virtual array aperture, is built. To leverage celebrated compressive sensing algorithms, the continuous parameter space has to be discretized to a number of fixed grid points, which inevitably incurs modeling error caused by off-grid gap. To remedy this issue, a block alternating optimization method is put forth that jointly estimates the sparse signal and refines the locations of grid points. Specifically, inspired by the majorization minimization, the proposed method iteratively minimizes a surrogate function majorizing the given objective function, where only a single block of variables are updated per iteration while the remaining ones are kept fixed. The proposed method features affordable computational complexity, and numerical tests corroborate its superior performance relative to existing alternatives in both overdetermined and underdetermined scenarios.
Dynamic Updating for L1 Minimization

Muhammad Salman Asif,Justin Romberg

DOI: https://doi.org/10.1109/JSTSP.2009.2039174

2009-06-29

Abstract:The theory of compressive sensing (CS) suggests that under certain conditions, a sparse signal can be recovered from a small number of linear incoherent measurements. An effective class of reconstruction algorithms involve solving a convex optimization program that balances the L1 norm of the solution against a data fidelity term. Tremendous progress has been made in recent years on algorithms for solving these L1 minimization programs. These algorithms, however, are for the most part static: they focus on finding the solution for a fixed set of measurements. In this paper, we will discuss "dynamic algorithms" for solving L1 minimization programs for streaming sets of measurements. We consider cases where the underlying signal changes slightly between measurements, and where new measurements of a fixed signal are sequentially added to the system. We develop algorithms to quickly update the solution of several different types of L1 optimization problems whenever these changes occur, thus avoiding having to solve a new optimization problem from scratch. Our proposed schemes are based on homotopy continuation, which breaks down the solution update in a systematic and efficient way into a small number of linear steps. Each step consists of a low-rank update and a small number of matrix-vector multiplications -- very much like recursive least squares. Our investigation also includes dynamic updating schemes for L1 decoding problems, where an arbitrary signal is to be recovered from redundant coded measurements which have been corrupted by sparse errors.

Information Theory
Gradient Projection with Approximate L0 Norm Minimization for Sparse Reconstruction in Compressed Sensing

Ziran Wei,Jianlin Zhang,Zhiyong Xu,Yongmei Huang,Yong Liu,Xiangsuo Fan

DOI: https://doi.org/10.3390/s18103373

IF: 3.9

2018-01-01

Sensors

Abstract:In the reconstruction of sparse signals in compressed sensing, the reconstruction algorithm is required to reconstruct the sparsest form of signal. In order to minimize the objective function, minimal norm algorithm and greedy pursuit algorithm are most commonly used. The minimum L1 norm algorithm has very high reconstruction accuracy, but this convex optimization algorithm cannot get the sparsest signal like the minimum L0 norm algorithm. However, because the L0 norm method is a non-convex problem, it is difficult to get the global optimal solution and the amount of calculation required is huge. In this paper, a new algorithm is proposed to approximate the smooth L0 norm from the approximate L2 norm. First we set up an approximation function model of the sparse term, then the minimum value of the objective function is solved by the gradient projection, and the weight of the function model of the sparse term in the objective function is adjusted adaptively by the reconstruction error value to reconstruct the sparse signal more accurately. Compared with the pseudo inverse of L2 norm and the L1 norm algorithm, this new algorithm has a lower reconstruction error in one-dimensional sparse signal reconstruction. In simulation experiments of two-dimensional image signal reconstruction, the new algorithm has shorter image reconstruction time and higher image reconstruction accuracy compared with the usually used greedy algorithm and the minimum norm algorithm.
Dual Alternating Direction Method of Multipliers for Inverse Imaging

Li Song,Zhou Ge,Edmund Y. Lam

DOI: https://doi.org/10.1109/tip.2022.3167915

IF: 10.6

2022-01-01

IEEE Transactions on Image Processing

Abstract:Inverse imaging covers a wide range of imaging applications, including super-resolution, deblurring, and compressive sensing. We propose a novel scheme to solve such problems by combining duality and the alternating direction method of multipliers (ADMM). In addition to a conventional ADMM process, we introduce a second one that solves the dual problem to find the estimated nontrivial lower bound of the objective function, and the related iteration results are used in turn to guide the primal iterations. We call this D-ADMM, and show that it converges to the global minimum when the regularization function is convex and the optimization problem has at least one optimizer. Furthermore, we show how the scheme can give rise to two specific algorithms, called D-ADMM-L2 and D-ADMM-TV, by having different regularization functions. We compare D-ADMM-TV with other methods on image super-resolution and demonstrate comparable or occasionally slightly better quality results. This paves the way of incorporating advanced operators and strategies designed for basic ADMM into the D-ADMM method as well to further improve the performances of those methods.
Nonconvex Optimization with Alternating Direction Method of Multipliers for Tensor-Based Compressed Sensing with Group Sparsity

Yong Li,Wenrui Dai,Hongkai Xiong

DOI: https://doi.org/10.1109/dcc.2017.22

2017-01-01

Abstract:Tensor-based compressive sensing (CS) can preserve the intrinsic multidimensional structures with a reduced computational complexity. However, its recovery performance is degraded by simple sparsity. This paper proposes an efficient recovery algorithm GT-ADMM to solve a nonconvex optimization problem for tensor-based CS with group sparsity. Group-sparse representations of tensors is derived based on the Kronecker product of adaptively trained basis. The proposed algorithm incorporates alternating direction method of multipliers (ADMM) to obtain desirable recovery quality with improved efficiency. It requires few degrees of freedom to achieve tensor recovery with a guarantee of minimized approximation error. Experimental results demonstrate that the proposed algorithm outperforms state-of-the-art tensor-based methods in compressive video sampling.
Zero-point attracting projection algorithm for sequential compressive sensing

Yang You,Jian Jin,Wei Duan,Ningning Liu,Yuantao Gu,Jian Yang

DOI: https://doi.org/10.48550/arXiv.1303.2251

2013-03-10

Abstract:Sequential Compressive Sensing, which may be widely used in sensing devices, is a popular topic of recent research. This paper proposes an online recovery algorithm for sparse approximation of sequential compressive sensing. Several techniques including warm start, fast iteration, and variable step size are adopted in the proposed algorithm to improve its online performance. Finally, numerical simulations demonstrate its better performance than the relative art.

Information Theory
Study on L1 over L2 Minimization for Nonnegative Signal Recovery

Min Tao,Xiao-Ping Zhang

DOI: https://doi.org/10.1007/s10915-023-02225-2

2023-01-01

Journal of Scientific Computing

Abstract:In this paper, we carry out a comprehensive study for the unconstrained L-1 over L-2 sparsity promoting model, widely used in the regime of coherent dictionaries for recovering nonnegative sparse signals. First, we prove the existence of global solutions. Second, we analyze the sparse property of any local minimizer of the L-1/L-2 model. This property serves as a certificate to rule out the nonlocal minimizers. Third, we focus on algorithmic development on the unconstrained model with nonnegative constraint. We derive an analytical solution for the proximal operator of the L-1/L-2 with nonnegative constraint. Then, we apply the alternating direction method of multipliers in a particular splitting way, referred to as ADMM(p)(+). We establish its global convergence to a d-stationary solution (sharpest stationary) by verifying the Lyapunov function with the Kurdyka-Lojasiewicz property instead of imposing. Extensive numerical simulations confirm the superiority of ADMM(p)(+) over the existing state-of-the-art methods in nonnegative sparse recovery.
Two-level ℓ1 Minimization for Compressed Sensing

Xiaolin Huang,Yipeng Liu,Lei Shi,Sabine Van Huffel,Johan A. K. Suykens

DOI: https://doi.org/10.1016/j.sigpro.2014.09.028

IF: 4.729

2015-01-01

Signal Processing

Abstract:Compressed sensing using ℓ1 minimization has been widely and successfully applied. To further enhance the sparsity, a non-convex and piecewise linear penalty is proposed. This penalty gives two different weights according to the order of the absolute value and hence is called the two-level ℓ1-norm. The two-level ℓ1-norm can be minimized by an iteratively reweighted ℓ1 method. Compared with some existing non-convex methods, the two-level ℓ1 minimization has similar sparsity and enjoys good convergence behavior. More importantly, the related soft thresholding algorithm has been established. The shrinkage operator for the two-level ℓ1-norm is not non-expansive and its convergence is proved by showing the monotone of the objective value in the iterations. In numerical experiments, the proposed algorithms achieve good sparse signal estimation performance, which makes the two-level ℓ1 minimization a promising technique for compressed sensing.

Primal and dual alternating direction algorithms for l1-l1-norm minimization problems in compressive sensing

Alternating Direction Algorithms for $\Ell_1$-Problems in Compressive Sensing

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

On the Null Space Property of L Q-Minimization for 0 < Q ≤ 1 in Compressed Sensing

A Primal Dual Active Set Algorithm with Continuation for Compressed Sensing

Alternating direction algorithms for $\ell_0$ regularization in compressed sensing.

A Primal Dual Active Set with Continuation Algorithm for the ℓ^0-Regularized Optimization Problem

A Primal Douglas–Rachford Splitting Method for the Constrained Minimization Problem in Compressive Sensing

A splitting primal-dual proximity algorithm for solving composite optimization problems

Study on L_1 over L_2 Minimization for Nonnegative Signal Recovery

Analysis of the Ratio of ℓ1 and ℓ2 Norms in Compressed Sensing

The variant of the iterative shrinkage-thresholding algorithm for minimization of the l 1 over l ∞ norms

Unified Analysis on L1 over L2 Minimization for Signal Recovery

A Block Alternating Optimization Method for Direction-of-Arrival Estimation with Nested Array.

Dynamic Updating for L1 Minimization

Gradient Projection with Approximate L0 Norm Minimization for Sparse Reconstruction in Compressed Sensing

Dual Alternating Direction Method of Multipliers for Inverse Imaging

Nonconvex Optimization with Alternating Direction Method of Multipliers for Tensor-Based Compressed Sensing with Group Sparsity

Zero-point attracting projection algorithm for sequential compressive sensing

Study on <i>L</i><sub>1</sub> over <i>L</i><sub>2</sub> Minimization for Nonnegative Signal Recovery

Two-level ℓ1 Minimization for Compressed Sensing