An Algorithm for Solving Scaled Total Least Squares Problems.

Sanzheng Qiao,Wei Xu,Yimin Wei

2008-01-01

Abstract:In this paper, we present a rankrevealing two-sided orthogonal decomposition method for solving the STLS problem. An error analysis of the algorithm is given. Our numerical experiments show that this algorithm computes the STLS solution as accurate as the SVD method with less computation.

What problem does this paper attempt to address?

An algorithm for solving rank-deficient scaled total least square problems

Wei Xu,Sanzheng Qiao,Yimin Wei

2004-01-01

Abstract:The scaled total least square (STLS) problem, introduced by B.D. Rao in 1997, unifies both the total least square (TLS) and the least square (LS) problems. The STLS problems can be solved by the singular value decomposition (SVD). In this paper, we give a rank-revealing two-sided orthogonal decomposition method for solving the STLS problem. An error analysis is presented. Our numerical experiments show that this algorithm computes the STLS solution as good as the SVD method with less computation.
A Note on the Scaled Total Least Squares Problem

Wei Xu,Sanzheng Qiao,Yimin Wei

DOI: https://doi.org/10.1016/j.laa.2007.03.015

IF: 1.307

2008-01-01

Linear Algebra and its Applications

Abstract:In this note, we present two results on the scaled total least squares problem. First, we discuss the relation between the scaled total least squares and the least squares problems. We derive an upper bound for the difference between the scaled total least squares solution and the least squares solution and establish a quantitative relation between the scaled total least squares residual and the least squares residual. Second, we give a perturbation analysis of the scaled total least squares problem. Numerical experiments in comparing our results with existing results are demonstrated.
Randomized algorithms for total least squares problems.

Pengpeng Xie,Hua Xiang,Yimin Wei

DOI: https://doi.org/10.1002/nla.2219

2019-01-01

Numerical Linear Algebra with Applications

Abstract:Motivated by the recently popular probabilistic methods for low-rank approximations and randomized algorithms for the least squares problems, we develop randomized algorithms for the total least squares problem with a single right-hand side. We present the Nystrom method for the medium-sized problems. For the large-scale and ill-conditioned cases, we introduce the randomized truncated total least squares with the known or estimated rank as the regularization parameter. We analyze the accuracy of the algorithm randomized truncated total least squares and perform numerical experiments to demonstrate the efficiency of our randomized algorithms. The randomized algorithms can greatly reduce the computational time and still maintain good accuracy with very high probability.
Perturbation Analysis and Randomized Algorithms for Large-Scale Total Least Squares Problems

Pengpeng Xie,Yimin Wei,Hua Xiang

DOI: https://doi.org/10.48550/arxiv.1401.6832

2014-01-01

Abstract:In this paper, we present perturbation analysis and randomized algorithms for the total least squares (TLS) problems. We derive the perturbation bound and check its sharpness by numerical experiments. Motivated by the recently popular probabilistic algorithms for low-rank approximations, we develop randomized algorithms for the TLS and the truncated total least squares (TTLS) solutions of large-scale discrete ill-posed problems, which can greatly reduce the computational time and still keep good accuracy.
On the Weighting Method for Mixed Least Squares–total Least Squares Problems

Qiaohua Liu,Minghui Wang

DOI: https://doi.org/10.1002/nla.2094

2017-01-01

Numerical Linear Algebra with Applications

Abstract:SummaryIt is well known that the standard algorithm for the mixed least squares–total least squares (MTLS) problem uses the QR factorization to reduce the original problem into a standard total least squares problem with smaller size, which can be solved based on the singular value decomposition (SVD). In this paper, the MTLS problem is proven to be closely related to a weighted total least squares problem with its error‐free columns multiplied by a large weighting factor. A criterion for choosing the weighting factor is given; and for the sake of stability in solving the MTLS problem, the Cholesky factorization‐based inverse (Cho‐INV) iteration and Rayleigh quotient iteration are also considered. For large‐scale MTLS problems, numerical tests show that Cho‐INV is superior to the standard QR‐SVD method, especially for the case with big gap between the desired and undesired singular values and the case when the coefficient matrix has much more error‐contaminated columns. Rayleigh quotient iteration behaves more efficient than QR‐SVD for most cases and fails occasionally, and in some cases, it converges much faster than Cho‐INV but still less efficient due to its higher computation cost.
Study on the two-stage least square algorithm and its performance for nonstationary stochastic system

HuaPing Dai,Jiawen Dong,Jixin Qian,Youxian Sun

1997-01-01

Abstract:In this paper, a novel, two-stage least square(TS-LS) algorithm is proposed by theoretical derivation for nonstationary dynamic systems with input slow time-varying disturbance, and we explain that TS-LS is an example of extended-matrix method. Numerical simulation results show the efficiency of extended-matrix methodis.
Research of Large-Scale Sparse Rsqp Algorithm Based on Line Search of Filter Method

Jiang Aipeng,Shao Zhijiang,Qian Jixin

DOI: https://doi.org/10.1109/wcica.2004.1340593

2004-01-01

Abstract:In process system optimization, it is common that problem has large number of equality constraints and bound constraints, and relatively few degrees of freedom. Reduced space algorithms are well suited for this category of problem. It is of great importance that sparse structure of specific problem be utilized to save memory and computation time. In this paper a sparse rSQP (reduced sequential quadratic programming) algorithm based on line search of filter method was presented for large-scale process system optimization. This algorithm solves a sequence of QP problems, which are constructed in reduced space by space decomposition and are relatively small in dimensions. Coordination decomposition was used to keep sparse structure of problems, and line search of filter method is incorporated to enhance efficiency. Numerical results in optimizing an example with variable dimension and a distillation column optimization problem demonstrate that the proposed algorithm is quite more efficient than the standard SQP algorithm.
A novel greedy Gauss-Seidel method for solving large linear least squares problem

Yanjun Zhang,Hanyu Li

DOI: https://doi.org/10.48550/arXiv.2004.03692

2020-04-08

Abstract:We present a novel greedy Gauss-Seidel method for solving large linear least squares problem. This method improves the greedy randomized coordinate descent (GRCD) method proposed recently by Bai and Wu [Bai ZZ, and Wu WT. On greedy randomized coordinate descent methods for solving large linear least-squares problems. Numer Linear Algebra Appl. 2019;26(4):1--15], which in turn improves the popular randomized Gauss-Seidel method. Convergence analysis of the new method is provided. Numerical experiments show that, for the same accuracy, our method outperforms the GRCD method in term of the computing time.

Numerical Analysis
Small Sample Statistical Condition Estimation for the Total Least Squares Problem

Huai-An Diao,Yimin Wei,Pengpeng Xie

DOI: https://doi.org/10.1007/s11075-016-0185-9

IF: 2.37

2016-01-01

Numerical Algorithms

Abstract:In this paper, under the genericity condition, we study the condition estimation of the total least squares (TLS) problem based on small sample condition estimation (SCE), which can be incorporated into the direct solver for the TLS problem via the singular value decomposition (SVD) of the augmented matrix [A, b]. Our proposed condition estimation algorithms are efficient for the small and medium size TLS problem because they utilize the computed SVD of [A, b] during the numerical solution to the TLS problem. Numerical examples illustrate the reliability of the algorithms. Both normwise and componentwise perturbations are considered. Moreover, structured condition estimations are investigated for the structured TLS problem.
Scalable Distributed Least Squares Algorithm for Linear Algebraic Equations via Scheduling

Shenyu Liu

2024-11-11

Abstract:In this work, we propose a novel discrete-time distributed algorithm for finding least squares solutions of linear algebraic equations with a scheduling protocol to further enhance its scalability. Each agent in the network is assumed to know some rows of the coefficient matrix and the corresponding entries in the observation vector. Unlike typical distributed algorithms, our approach considers communication bandwidth limits, allowing agents to transmit only a portion of their ``guessed" solution, independent of its dimension. A periodic scheduling protocol determines which portion is transmitted at each iteration. Assuming a small fixed step size and a diagonalizable algorithm matrix, we prove that agents' ``guessed" solutions converge exponentially to a least squares solution. For cases where the observation vectors are time-varying, a modified algorithm guarantees practical convergence, with tracking error bounded by the single-step variation in the observation vector. Simulations on both small-scale and medium-scale examples validate the feasibility of our algorithms.

Systems and Control
Randomized symmetric Gauss-Seidel method for solving linear least squares problems

Fan Sha,Jianbing Zhang

DOI: https://doi.org/10.3934/math.2024848

2024-01-01

AIMS Mathematics

Abstract:We introduced a random symmetric Gauss-Seidel (RSGS) method, which was designed to handle large scale linear least squares problems involving tall coefficient matrices. This RSGS method projected the approximate residual onto the subspace spanned by two symmetric columns at each iteration. These columns were sampled from the coefficient matrix based on an effective probability criterion. Our theoretical analysis indicated that RSGS converged when the coefficient matrix had full column rank. Furthermore, numerical experiments demonstrated that RSGS outperformed the baseline algorithms in terms of iteration steps and CPU time.

mathematics, applied
Scalable Distributed Least Square Algorithms for Large-Scale Linear Equations Via an Optimization Approach

Yi Huang,Ziyang Meng,Jian Sun

DOI: https://doi.org/10.1016/j.automatica.2022.110572

IF: 6.4

2022-01-01

Automatica

Abstract:This paper proposes scalable distributed least square algorithms for solving large-scale linear algebraic equations via multi-agent network. We consider two kinds of different decomposition structures of A x = b, in which each agent only knows a sub-block of linear equation instead of the complete row or column information in the existing results. By introducing one extra variable, the least square problem of linear equation can be transformed into a distributed constrained optimization problem. Based on the augmented Lagrange method, two continuous-time scalable distributed primal–dual algorithms are developed, in which all the states of agents have fewer dimension and can be even scalars. Subsequently, the discrete-time distributed algorithms with constant step-sizes are proposed by using the Euler discretization method, and the explicit bounds of constant step-sizes are provided. Based on the KKT condition and Lyapunov stability, we show that the proposed continuous-time and discrete-time distributed algorithms collaboratively obtain a least square solution with a linear convergence speed, and also own an additional property that verifies whether the obtained solution is an exact solution. Finally, some simulation examples are carried out to verify the effectiveness of the proposed algorithms.
Condition Numbers of the Multidimensional Total Least Squares Problems Having More Than One Solution

Lingsheng Meng,Bing Zheng,Yimin Wei

DOI: https://doi.org/10.1007/s11075-019-00785-9

IF: 2.37

2019-01-01

Numerical Algorithms

Abstract:Recently, the condition numbers of the total least squares (TLS) problems having a unique solution have been studied at length in Zheng et al. (SIAM J. Matrix Anal. Appl. 38 : 924–948, 2017 ). However, it is known that the TLS problem may have no solution, and even if an existing solution, it may not be unique. As a continuation of their work, in this paper, we investigate the condition numbers of the minimum Frobenius norm solution of the (multidimensional) TLS problem when having more than one solution. The tight and computable upper bound estimates of the normwise, mixed, and componentwise condition numbers are respectively derived. Some numerical experiments are performed to illustrate the tightness of these upper bounds.
On an Iterative Method for Solving the Least Squares Problem of Rank-Deficient Systems

Le-ping Sun,Yi-min Wei,Jie-yong Zhou

DOI: https://doi.org/10.1080/00207160.2014.900173

2014-01-01

Abstract:In this paper, we first show an iterative method for finding the least squares (LS) solution to the inconsistent system Ax=b, where A is an mxn matrix of rank r. The method is an iteration scheme for consistent system of linear equations M=(b)over-cap which is an augmented system associated with Ax=b. It denotes that under some conditions, the sequence, (0), (1), (2), horizontal ellipsis , converges to the LS solution of the system Ax=b for every initial vector (0), where (M+gamma E)(i)=gamma Ei-1+(b)over-cap, for i=1, 2, horizontal ellipsis . In our numerical test, we propose to find E without using decomposition methods. The improved timings are shown with matrices of substantial size.
Roundoff Error Analysis of an Algorithm Based on Householder Bidiagonalization for Total Least Squares Problems

Zhanshan Yang,Xilan Liu

DOI: https://doi.org/10.3390/math9202550

IF: 2.4

2021-10-12

Mathematics

Abstract:For large-scale problems, how to establish an algorithm with high accuracy and stability is particularly important. In this paper, the Householder bidiagonalization total least squares (HBITLS) algorithm and nonlinear iterative partial least squares for total least squares (NIPALS-TLS) algorithm were established, by which the same approximate TLS solutions was obtained. In addition, the propagation of the roundoff error for the process of the HBITLS algorithm was analyzed, and the mixed forward-backward stability of these two algorithms was proved. Furthermore, an upper bound of roundoff error was derived, which presents a more detailed and clearer approximation of the computed solution.

mathematics
Calibrating Least Squares Semidefinite Programming with Equality and Inequality Constraints

Yan Gao,Defeng Sun

DOI: https://doi.org/10.1137/080727075

IF: 1.908

2010-01-01

SIAM Journal on Matrix Analysis and Applications

Abstract:In this paper, we consider the least squares semidefinite programming with a large number of equality and inequality constraints. One difficulty in finding an efficient method for solving this problem is due to the presence of the inequality constraints. In this paper, we propose to overcome this difficulty by reformulating the problem as a system of semismooth equations with two level metric projection operators. We then design an inexact smoothing Newton method to solve the resulting semismooth system. At each iteration, we use the BiCGStab iterative solver to obtain an approximate solution to the generated smoothing Newton linear system. Our numerical experiments confirm the high efficiency of the proposed method.

mathematics, applied
On adaptive stochastic extended iterative methods for solving least squares

Yun Zeng,Deren Han,Yansheng Su,Jiaxin Xie

2024-05-29

Abstract:In this paper, we propose a novel adaptive stochastic extended iterative method, which can be viewed as an improved extension of the randomized extended Kaczmarz (REK) method, for finding the unique minimum Euclidean norm least-squares solution of a given linear system. In particular, we introduce three equivalent stochastic reformulations of the linear least-squares problem: stochastic unconstrained and constrained optimization problems, and the stochastic multiobjective optimization problem. We then alternately employ the adaptive variants of the stochastic heavy ball momentum (SHBM) method, which utilize iterative information to update the parameters, to solve the stochastic reformulations. We prove that our method converges linearly in expectation, addressing an open problem in the literature related to designing theoretically supported adaptive SHBM methods. Numerical experiments show that our adaptive stochastic extended iterative method has strong advantages over the non-adaptive one.

Numerical Analysis
Solving Large Rank-Deficient Linear Least-Squares Problems on Shared-Memory CPU Architectures and GPU Architectures

Mónica Chillarón,Gregorio Quintana-Ortí,Vicente Vidal,Per-Gunnar Martinsson

DOI: https://doi.org/10.48550/arXiv.2408.05238

2024-08-06

Abstract:Solving very large linear systems of equations is a key computational task in science and technology. In many cases, the coefficient matrix of the linear system is rank-deficient, leading to systems that may be underdetermined, inconsistent, or both. In such cases, one generally seeks to compute the least squares solution that minimizes the residual of the problem, which can be further defined as the solution with smallest norm in cases where the coefficient matrix has a nontrivial nullspace. This work presents several new techniques for solving least squares problems involving coefficient matrices that are so large that they do not fit in main memory. The implementations include both CPU and GPU variants. All techniques rely on complete orthogonal decompositions that guarantee that both conditions of a least squares solution are met, regardless of the rank properties of the matrix. Specifically, they rely on the recently proposed "randUTV" algorithm that is particularly effective in strongly communication-constrained environments. A detailed precision and performance study reveals that the new methods, that operate on data stored on disk, are competitive with state-of-the-art methods that store all data in main memory.

Distributed, Parallel, and Cluster Computing,Performance
Condition numbers for the truncated total least squares problem and their estimations

Qing-Le Meng,Huai-An Diao,Zheng-Jian Bai

DOI: https://doi.org/10.48550/arXiv.2004.12082

2020-04-29

Abstract:In this paper, we present explicit expressions for the mixed and componentwise condition numbers of the truncated total least squares (TTLS) solution of $A\boldsymbol{x} \approx \boldsymbol{b} $ under the genericity condition, where $A$ is a $m\times n$ real data matrix and $\boldsymbol{b}$ is a real $m$-vector. Moreover, we reveal that normwise, componentwise and mixed condition numbers for the TTLS problem can recover the previous corresponding counterparts for the total least squares (TLS) problem when the truncated level of for the TTLS problem is $n$. When $A$ is a structured matrix, the structured perturbations for the structured truncated TLS (STTLS) problem are investigated and the corresponding explicit expressions for the structured normwise, componentwise and mixed condition numbers for the STTLS problem are obtained. Furthermore, the relationships between the structured and unstructured normwise, componentwise and mixed condition numbers for the STTLS problem are studied. Based on small sample statistical condition estimation (SCE), reliable condition estimation algorithms for both unstructured and structured normwise, mixed and componentwise are devised, which utilize the SVD of the augmented matrix $[A~\boldsymbol{b} ]$. The efficient proposed condition estimation algorithms can be integrated into the SVD-based direct solver for the small and medium size TTLS problem to give the error estimation for the numerical TTLS solution. Numerical experiments are reported to illustrate the reliability of the proposed estimation algorithms, which coincide with our theoretical results.

Numerical Analysis
Single-pass randomized QLP decomposition for low-rank approximation

Huan Ren,Zheng-Jian Bai

DOI: https://doi.org/10.48550/arXiv.2011.06855

2020-11-27

Abstract:The QLP decomposition is one of the effective algorithms to approximate singular value decomposition (SVD) in numerical linear algebra. In this paper, we propose some single-pass randomized QLP decomposition algorithms for computing the low-rank matrix approximation. Compared with the deterministic QLP decomposition, the complexity of the proposed algorithms does not increase significantly and the system matrix needs to be accessed only once. Therefore, our algorithms are very suitable for a large matrix stored outside of memory or generated by stream data. In the error analysis, we give the bounds of matrix approximation error and singular value approximation error. Numerical experiments also reported to verify our results.

Numerical Analysis

An Algorithm for Solving Scaled Total Least Squares Problems.

An algorithm for solving rank-deficient scaled total least square problems

A Note on the Scaled Total Least Squares Problem

Randomized algorithms for total least squares problems.

Perturbation Analysis and Randomized Algorithms for Large-Scale Total Least Squares Problems

On the Weighting Method for Mixed Least Squares–total Least Squares Problems

Study on the two-stage least square algorithm and its performance for nonstationary stochastic system

Research of Large-Scale Sparse Rsqp Algorithm Based on Line Search of Filter Method

A novel greedy Gauss-Seidel method for solving large linear least squares problem

Small Sample Statistical Condition Estimation for the Total Least Squares Problem

Scalable Distributed Least Squares Algorithm for Linear Algebraic Equations via Scheduling

Randomized symmetric Gauss-Seidel method for solving linear least squares problems

Scalable Distributed Least Square Algorithms for Large-Scale Linear Equations Via an Optimization Approach

Condition Numbers of the Multidimensional Total Least Squares Problems Having More Than One Solution

On an Iterative Method for Solving the Least Squares Problem of Rank-Deficient Systems

Roundoff Error Analysis of an Algorithm Based on Householder Bidiagonalization for Total Least Squares Problems

Calibrating Least Squares Semidefinite Programming with Equality and Inequality Constraints

On adaptive stochastic extended iterative methods for solving least squares

Solving Large Rank-Deficient Linear Least-Squares Problems on Shared-Memory CPU Architectures and GPU Architectures

Condition numbers for the truncated total least squares problem and their estimations

Single-pass randomized QLP decomposition for low-rank approximation