Convergence Analysis of l0-RLS Adaptive Filter

B. K. Das,S. Mukhopadhyay,M. Chakraborty

DOI: https://doi.org/10.48550/arXiv.1710.09259

2017-10-16

Abstract:This paper presents first and second order convergence analysis of the sparsity aware l0-RLS adaptive filter. The theorems 1 and 2 state the steady state value of mean and mean square deviation of the adaptive filter weight vector.

Signal Processing,Machine Learning

What problem does this paper attempt to address?

CONVERGENCE AND STABILITY OF ITERATIVELY REWEIGHTED LEAST SQUARES FOR LOW-RANK MATRIXRECOVERY

Yun Cai,Song Li

DOI: https://doi.org/10.3934/ipi.2017030

2017-01-01

Inverse Problems and Imaging

Abstract:In this paper, we study the theoretical properties of iteratively reweighted least squares algorithm for recovering a matrix (IRLS-M for short) from noisy linear measurements. The IRLS-M was proposed by Fornasier et al. (2011) [ 17 ] for solving nuclear norm minimization and by Mohan et al. (2012) [ 31 ] for solving Schatten- \begin{document}$p$\end{document} (quasi) norm minimization ( \begin{document}$0 ) in noiseless case, based on the iteratively reweighted least squares algorithm for sparse signal recovery (IRLS for short) (Daubechies et al., 2010) [ 15 ], and numerical experiments have been given to show its efficiency (Fornasier et al. and Mohan et al.) [ 17 ], [ 31 ]. In this paper, we focus on providing convergence and stability analysis of iteratively reweighted least squares algorithm for low-rank matrix recovery in the presence of noise. The convergence of IRLS-M is proved strictly for all \begin{document}$0 . Furthermore, when the measurement map \begin{document}$\mathcal{A}$\end{document} satisfies the matrix restricted isometry property (M-RIP for short), we show that the IRLS-M is stable for \begin{document}$0 . Specially, when \begin{document}$p=1$\end{document} , we prove that the M-RIP constant \begin{document}$δ_{2r} is sufficient for IRLS-M to recover an unknown (approximately) low rank matrix with an error that is proportional to the noise level. The simplicity of IRLS-M, along with the theoretical guarantees provided in this paper, make a compelling case for its adoption as a standard tool for low rank matrix recovery.
Steady-state performance analysis for adaptive filters with error nonlinearities

Bin Lin,Rongxi He, Liming Song,Baisuo Wang

DOI: https://doi.org/10.1109/ICASSP.2009.4960278

2009-01-01

Abstract:A unified approach to the steady-state mean square error (MSE) and tracking performance analyses for real and complex adaptive filtes with error nonlinearities is developed. Some general clofied-form analytical expressions for the steady-state performances are given. Our analyses are based on Taylor series expansion and and so-called complex Brandwood-form series expansion (BSE). Under these general explicit expressions, some well-known adaptive filters can be viewed as special cases. In addition, the closed-form analytical expressions for the steady-state performance for real and complex least-mean p-power (LMP) algorithm with different choices of parameter p are also given. A mass of simulations show the accuration of our analyses.
Convergence Analysis of the Modified Frequency-Domain Block LMS Algorithm with Guaranteed Optimal Steady State Performance

Jing Lu,Xiaojun Qiu,Haishan Zou

DOI: https://doi.org/10.1016/j.sigpro.2016.10.004

IF: 4.729

2016-01-01

Signal Processing

Abstract:Although the bin-normalized frequency domain block LMS (NFBLMS) algorithm has theoretically very fast convergence speed, it suffers from convergence to a biased steady state solution when causality is not met or when the adaptive filter is of deficient length. A modified FBLMS (MFBLMS) algorithm with guaranteed optimal steady state performance has been proposed, but the theoretical analysis on its convergence properties has not been presented. This paper analyzes the convergence behavior of the algorithm by using the theory of asymptotically equivalent matrices. The eigenvalues of the matrix controlling the convergence behavior is proven to have the tendency to be equally distributed, and a theoretical eigenvalue spread is derived based on the first-order autoregressive (AR) signal model, which is significantly lower than that of the time domain LMS (TDLMS) algorithm. Therefore the convergence speed of the MFBLMS is significantly higher than that of the TDLMS algorithm for colored reference signal. Simulations are carried out to validate the convergence behavior predicted from the theoretical analysis.
On the Mean-Square Performance of the Constrained LMS Algorithm

Reza Arablouei,Kutluyıl Doğançay,Stefan Werner

DOI: https://doi.org/10.48550/arXiv.1412.2424

2015-02-25

Abstract:The so-called constrained least mean-square algorithm is one of the most commonly used linear-equality-constrained adaptive filtering algorithms. Its main advantages are adaptability and relative simplicity. In order to gain analytical insights into the performance of this algorithm, we examine its mean-square performance and derive theoretical expressions for its transient and steady-state mean-square deviation. Our methodology is inspired by the principle of energy conservation in adaptive filters. Simulation results corroborate the accuracy of the derived formula.

Systems and Control
Sparse Normalized Subband Adaptive Filter Algorithm with L0-Norm Constraint

Yi Yu,Haiquan Zhao,Badong Chen

DOI: https://doi.org/10.1016/j.jfranklin.2016.09.022

IF: 4.246

2016-01-01

Journal of the Franklin Institute

Abstract:In order to improve the filter׳s performance when identifying sparse system, this paper develops two sparse-aware algorithms by incorporating the l0-norm constraint of the weight vector into the conventional normalized subband adaptive filter (NSAF) algorithm. The first algorithm is obtained from the principle of the minimum perturbation; and the second one is based on the gradient descent principle. The resulting algorithms have almost the same convergence and steady-state performance while the latter saves computational complexity. What׳s more, the performance of both algorithms is analyzed by resorting to some assumptions commonly used in the analyses of adaptive algorithms. Simulation results in the context of sparse system identification not only demonstrate the effectiveness of the proposed algorithms, but also verify the theoretical analyses.
Convergence Analysis of a Deficient-Length LMS Filter and Optimal-Length Sequence to Model Exponential Decay Impulse Response

YT Gu,K Tang,HJ Cui,W Du

DOI: https://doi.org/10.1109/lsp.2002.806704

2003-01-01

IEEE Signal Processing Letters

Abstract:This letter analyzes the mean-square convergence of a deficient-length least mean-square adaptive filter whose length is less than that of the unknown system and proves that filter length and the envelope of impulse response are the important factors in convergence rate control. For the impulse response with an exponential decay envelope, which covers a large set of physical systems, e.g., acoustic echo path, an optimal filter length sequence is figured out to achieve the fastest convergence. The simulations of an exact exponential decay envelope and of a real-life echo path in a car environment are performed via computer, and the results validate our analysis well.
On Parameter Convergence of Nonlinearly Parameterized Adaptive Systems: Analysis Via Contraction and First Lyapunov's Methods

Lei Wang,Romeo Ortega,Hongye Su,Zhitao Liu,Weihua Xu

DOI: https://doi.org/10.1109/cdc.2014.7039437

2014-01-01

Abstract:Adaptive systems have been traditionally analyzed using classical second Lyapunov's method and/or passivity theory. These powerful tools permit the establishment of conditions for convergence and stability (asymptotic and ℒ2) of adaptive estimators and controllers. A natural, alternative framework to analyze adaptive systems invokes the notion of (flow) contraction, which imposes that some distance between any pair of solutions is monotonically decreasing with time. In this paper we investigate the contraction properties of adaptive systems designed following the principles of immersion and invariance. In particular, we use contraction theory and a recent extension of first Lyapunov's method to give conditions under which adaptive estimators and controllers-designed for nonlinearly parameterized, nonlinear systems-ensure exponential parameter convergence.
Liapunov Analysis and Design of Linearly Constrained Adaptive Filtering Algorithms

Oumar Diene,Amit Bhaya

2007-01-01

Abstract:Multi-user mobile communication systems use linearly constrained adaptive fllters for a blind adaptive interference cancelation and/or adaptive beamforming. Con- jugate gradient (CG) techniques have been proposed, in the literature, for solving the adaptive flltering and the linearly constrained adaptive flltering problems. In adaptive flltering, the sample-by-sample update of the correlation matrix and the cross-correlation vector causes a loss of the residue orthogonality in a modifled on- line algorithm, which, in turn, results in loss of convergence and an increase of the fllter quadratic mean error. This paper extends a recently proposed optimality and convergence proof of the degenerated CG method to the case of linearly constrained adaptive flltering, and proposes a constrained Steepest Descent (CSD) method. The proposed algorithms are applied to a single-user detection in a DS-CDMA mobile communication system and shown to have performance comparable to those of al- gorithms proposed earlier without formal convergence proofs.
Performance Analysis of $l_0$ Norm Constraint Least Mean Square Algorithm

Guolong Su,Jian Jin,Yuantao Gu,Jian Wang

DOI: https://doi.org/10.1109/tsp.2012.2184537

IF: 4.875

2012-01-01

IEEE Transactions on Signal Processing

Abstract:As one of the recently proposed algorithms for sparse system identification, $l_0$ norm constraint Least Mean Square ($l_0$-LMS) algorithm modifies the cost function of the traditional method with a penalty of tap-weight sparsity. The performance of $l_0$-LMS is quite attractive compared with its various precursors. However, there has been no detailed study of its performance. This paper presents all-around and throughout theoretical performance analysis of $l_0$-LMS for white Gaussian input data based on some reasonable assumptions. Expressions for steady-state mean square deviation (MSD) are derived and discussed with respect to algorithm parameters and system sparsity. The parameter selection rule is established for achieving the best performance. Approximated with Taylor series, the instantaneous behavior is also derived. In addition, the relationship between $l_0$-LMS and some previous arts and the sufficient conditions for $l_0$-LMS to accelerate convergence are set up. Finally, all of the theoretical results are compared with simulations and are shown to agree well in a large range of parameter setting.
Performance Analysis of the Gradient Comparator LMS Algorithm

Bijit Kumar Das,Mrityunjoy Chakraborty

DOI: https://doi.org/10.48550/arXiv.1605.02877

2016-05-10

Abstract:The sparsity-aware zero attractor least mean square (ZA-LMS) algorithm manifests much lower misadjustment in strongly sparse environment than its sparsity-agnostic counterpart, the least mean square (LMS), but is shown to perform worse than the LMS when sparsity of the impulse response decreases. The reweighted variant of the ZA-LMS, namely RZA-LMS shows robustness against this variation in sparsity, but at the price of increased computational complexity. The other variants such as the l 0 -LMS and the improved proportionate normalized LMS (IPNLMS), though perform satisfactorily, are also computationally intensive. The gradient comparator LMS (GC-LMS) is a practical solution of this trade-off when hardware constraint is to be considered. In this paper, we analyse the mean and the mean square convergence performance of the GC-LMS algorithm in detail. The analyses satisfactorily match with the simulation results.

Information Theory,Machine Learning
Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem

Tang, H.J.,Tan, K.C.,Zhang, Y.

DOI: https://doi.org/10.1109/IJCNN.2002.1007530

2002-01-01

Abstract:We study the convergence of a class of discrete recurrent neural networks to solve linear variational inequality problem (LVIP). LVIP has important applications in engineering and economics. Not only the network's exponential convergence for the case of positive definite matrix is proved, but its global convergence for positive semidefinite matrix is also proved. Conditions are derived to guarantee the convergences of the network. Comprehensive examples are discussed and simulated to illustrate the results
Sparse LMS for System Identification

Yilun Chen,Yuantao Gu,Alfred O. Hero

DOI: https://doi.org/10.1109/icassp.2009.4960286

2009-01-01

Abstract:We propose a new approach to adaptive system identification when the system model is sparse. The approach applies ℓ1 relaxation, common in compressive sensing, to improve the performance of LMS-type adaptive methods. This results in two new algorithms, the zero-attracting LMS (ZA-LMS) and the reweighted zero-attracting LMS (RZA-LMS). The ZA-LMS is derived via combining a ℓ1 norm penalty on the coefficients into the quadratic LMS cost function, which generates a zero attractor in the LMS iteration. The zero attractor promotes sparsity in taps during the filtering process, and therefore accelerates convergence when identifying sparse systems. We prove that the ZA-LMS can achieve lower mean square error than the standard LMS. To further improve the filtering performance, the RZA-LMS is developed using a reweighted zero attractor. The performance of the RZA-LMS is superior to that of the ZA-LMS numerically. Experiments demonstrate the advantages of the proposed filters in both convergence rate and steady-state behavior under sparsity assumptions on the true coefficient vector. The RZA-LMS is also shown to be robust when the number of non-zero taps increases.
Stable Total Least Mean Square Adaptive Filter Algorithm

孔祥玉,魏瑞轩,韩崇昭,马红光

DOI: https://doi.org/10.3321/j.issn:0253-987X.2004.08.016

2004-01-01

Abstract:Aiming at the filter problem that the input and output signal are both corrupted by noise, a stable total least mean square (LMS) adaptive algorithm was proposed. Taking the sum of Rayleigh quotient of the augmented weight vectors of the system and a constraint to the last element of the augmented weight vectors via a LaGrange multiplier as an overall cost function, using the steepest descent principle, the adaptive updating formula of the weight vector was derived, the stability of the algorithm was analyzed, and the range of the learning factor to which the stability was guaranteed was educed. The proposed algorithm is stable and its computation complexity is lower, which can be realized neither calculating the squares root, nor normalization. The simulation results show that the convergence performance, the robust against noise and the convergence precision of the proposed algorithm are remarkably higher than those of other total LMS algorithms.
Convergence and stability of iteratively reweighted least squares for low-rank matrix recovery

Song Li,Yun Cai

DOI: https://doi.org/10.3934/IPI.2017030

2017-06-01

Abstract:In this paper, we study the theoretical properties of iteratively reweighted least squares algorithm for recovering a matrix (IRLS-M for short) from noisy linear measurements. The IRLS-M was proposed by Fornasier et al. (2011) [ 17 ] for solving nuclear norm minimization and by Mohan et al. (2012) [ 31 ] for solving Schatten- \begin{document}$p$\end{document} (quasi) norm minimization ( \begin{document}$0 ) in noiseless case, based on the iteratively reweighted least squares algorithm for sparse signal recovery (IRLS for short) (Daubechies et al., 2010) [ 15 ], and numerical experiments have been given to show its efficiency (Fornasier et al. and Mohan et al.) [ 17 ], [ 31 ]. In this paper, we focus on providing convergence and stability analysis of iteratively reweighted least squares algorithm for low-rank matrix recovery in the presence of noise. The convergence of IRLS-M is proved strictly for all \begin{document}$0 . Furthermore, when the measurement map \begin{document}$\mathcal{A}$\end{document} satisfies the matrix restricted isometry property (M-RIP for short), we show that the IRLS-M is stable for \begin{document}$0 . Specially, when \begin{document}$p=1$\end{document} , we prove that the M-RIP constant \begin{document}$δ_{2r} is sufficient for IRLS-M to recover an unknown (approximately) low rank matrix with an error that is proportional to the noise level. The simplicity of IRLS-M, along with the theoretical guarantees provided in this paper, make a compelling case for its adoption as a standard tool for low rank matrix recovery.

Mathematics,Computer Science
SPARLS: A Low Complexity Recursive $\mathcal{L}_1$-Regularized Least Squares Algorithm

Behtash Babadi,Nicholas Kalouptsidis,Vahid Tarokh

DOI: https://doi.org/10.48550/arXiv.0901.0734

2009-01-07

Abstract:We develop a Recursive $\mathcal{L}_1$-Regularized Least Squares (SPARLS) algorithm for the estimation of a sparse tap-weight vector in the adaptive filtering setting. The SPARLS algorithm exploits noisy observations of the tap-weight vector output stream and produces its estimate using an Expectation-Maximization type algorithm. Simulation studies in the context of channel estimation, employing multi-path wireless channels, show that the SPARLS algorithm has significant improvement over the conventional widely-used Recursive Least Squares (RLS) algorithm, in terms of both mean squared error (MSE) and computational complexity.

Information Theory
Convergence Speed of a Dynamical System for Sparse Recovery

Aurèle Balavoine,Christopher J. Rozell,Justin Romberg

DOI: https://doi.org/10.1109/TSP.2013.2271482

2013-05-21

Abstract:This paper studies the convergence rate of a continuous-time dynamical system for L1-minimization, known as the Locally Competitive Algorithm (LCA). Solving L1-minimization} problems efficiently and rapidly is of great interest to the signal processing community, as these programs have been shown to recover sparse solutions to underdetermined systems of linear equations and come with strong performance guarantees. The LCA under study differs from the typical L1 solver in that it operates in continuous time: instead of being specified by discrete iterations, it evolves according to a system of nonlinear ordinary differential equations. The LCA is constructed from simple components, giving it the potential to be implemented as a large-scale analog circuit. The goal of this paper is to give guarantees on the convergence time of the LCA system. To do so, we analyze how the LCA evolves as it is recovering a sparse signal from underdetermined measurements. We show that under appropriate conditions on the measurement matrix and the problem parameters, the path the LCA follows can be described as a sequence of linear differential equations, each with a small number of active variables. This allows us to relate the convergence time of the system to the restricted isometry constant of the matrix. Interesting parallels to sparse-recovery digital solvers emerge from this study. Our analysis covers both the noisy and noiseless settings and is supported by simulation results.

Dynamical Systems,Classical Analysis and ODEs,Optimization and Control
A New SPNLMS Algorithm Based on l∞-LMS

Bo ZHANG,Li TONG,Lingyun ZHOU,Chang CHEN

DOI: https://doi.org/10.3969/j.issn.1003-3106.2014.11.05

2014-01-01

Radio Engineering

Abstract:The SPNLMS algorithm used by sparse impulse response adaptive filter with a long order has a multiplication operation between every coefficient of the filter and its scale factor,which products a non-negligible amount of calculation.The shortcoming can be overcome by the l∞-SPNLMS algorithm proposed by combing l∞-LMS algorithm and SPNLMS algorithm. By reducing the amount of calculation,the l∞-LMS algorithm can save computing time,hardware resource and power consumption in project realization.The theoret-ical analysis and numerical simulation demonstrate that the l∞-SPNLMS algorithm has the characteristic of a fast convergence rate with sufficient convergence precision and a small amount of calculation in the early process of convergence.
Regularized Least-Mean-Square Algorithms

Yilun Chen,Yuantao Gu,Alfred O. Hero

2010-01-01

Abstract:We consider adaptive system identification problems with convex constraints and propose a family of regularized Least-Mean-Square (LMS) algorithms. We show that with a properly selected regularization parameter the regularized LMS provably dominates its conventional counterpart in terms of mean square deviations. We establish simple and closed-form expressions for choosing this regularization parameter. For identifying an unknown sparse system we propose sparse and group-sparse LMS algorithms, which are special examples of the regularized LMS family. Simulation results demonstrate the advantages of the proposed filters in both convergence rate and steady-state error under sparsity assumptions on the true coefficient vector.
Lms and Frls 2-D Lattice Filters

X LIU,M NAJIM,M JANATI,H YOULAL

DOI: https://doi.org/10.1016/s1474-6670(17)50781-8

1992-01-01

IFAC Proceedings Volumes

Abstract:In this paper we examine the two-dimensional extensions of some popular adaptive lattice filters. Adaptation is based on either the normalized least mean squares "NLMS" or the fast recursive least squares "FRLS" algorithms. These algorithms can update the filter coefficients in growing-order form with a moderate computational complexity. Derivation of the 2-D adaptive lattice "TDAL-NLMS" algorithm is straightforward, ones the 2-D lattice structure has been appropriately defined. The causality is first specified by associating the past 2-D data with the region of support. Then by appropriately defining the partial order of 2-D data, order recursion relations are obtained. In case of the FRLS algorithm we also make use of the 1-D multichannel analogy to derive the order recursion relations and the shift invariance property. The corresponding 2-D prediction problem is solved using the geometrical approaches of the vector space and the orthogonal projection. To compare the performances of these algorithms we consider first an adaptive estimation problem, then the implementation of the 2-D joint process lattice "TDJPL" filters for adaptive restoration of noisy images. Performance evaluation of LMS and FRLS based noise cancelling filters was done using artificially degraded image data at different signal to noise "SNR" ratio. The results show that higher improvement with the FRLS lattice filters can be expected .
Performance Analysis of L-0-Lms with Gaussian Input Signal

Guolong Su,Jian Jin,Yuantao Gu

DOI: https://doi.org/10.1109/icosp.2010.5655179

2010-01-01

Abstract:Sparse signal processing has attracted much attention in recent years. l0-LMS, which inserts a penalty of approximated l0 norm in the cost function of standard LMS algorithm, is one of the recently proposed sparse system identification algorithms. Numerical simulation results and intuitive explanations demonstrate that l0-LMS has rather small steady-state misalignment and fast convergence rate, especially with selected parameters, compared to its various precursors. In this paper, the mean square performance of l0-LMS is theoretically analyzed based on uncorrelated Gaussian input, independence assumption, and some other reasonable assumptions. We deduce the convergence condition on step-size, the steady-state mean square deviation, as well as the criterion on parameters selection. Finally, computer simulations verified the above theoretical results and confirmed the adopted assumptions hold well.

Convergence Analysis of l0-RLS Adaptive Filter

CONVERGENCE AND STABILITY OF ITERATIVELY REWEIGHTED LEAST SQUARES FOR LOW-RANK MATRIXRECOVERY

Steady-state performance analysis for adaptive filters with error nonlinearities

Convergence Analysis of the Modified Frequency-Domain Block LMS Algorithm with Guaranteed Optimal Steady State Performance

On the Mean-Square Performance of the Constrained LMS Algorithm

Sparse Normalized Subband Adaptive Filter Algorithm with L0-Norm Constraint

Convergence Analysis of a Deficient-Length LMS Filter and Optimal-Length Sequence to Model Exponential Decay Impulse Response

On Parameter Convergence of Nonlinearly Parameterized Adaptive Systems: Analysis Via Contraction and First Lyapunov's Methods

Liapunov Analysis and Design of Linearly Constrained Adaptive Filtering Algorithms

Performance Analysis of $l_0$ Norm Constraint Least Mean Square Algorithm

Performance Analysis of the Gradient Comparator LMS Algorithm

Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem

Sparse LMS for System Identification

Stable Total Least Mean Square Adaptive Filter Algorithm

Convergence and stability of iteratively reweighted least squares for low-rank matrix recovery

SPARLS: A Low Complexity Recursive $\mathcal{L}_1$-Regularized Least Squares Algorithm

Convergence Speed of a Dynamical System for Sparse Recovery

A New SPNLMS Algorithm Based on l∞-LMS

Regularized Least-Mean-Square Algorithms

Lms and Frls 2-D Lattice Filters

Performance Analysis of L-0-Lms with Gaussian Input Signal