Abstract:We introduce innovative algorithms for computing exact or approximate (minimum-norm) solutions to $Ax=b$ or the {\it normal equation} $A^TAx=A^Tb$, where $A$ is an $m \times n$ real matrix of arbitrary rank. We present more efficient algorithms when $A$ is symmetric PSD. First, we introduce the {\it Triangle Algorithm} (TA), a {\it convex-hull membership} algorithm that given $b_k=Ax_k$ in the ellipsoid $E_{A,\rho}=\{Ax: \Vert x \Vert \leq \rho\}$, it either computes an improved approximation $b_{k+1}=Ax_{k+1}$ or proves $b \not \in E_{A,\rho}$. We then give a dynamic variant of TA, the {\it Centering Triangle Algorithm} (CTA), generating residual, $r_k=b -Ax_k$ via the iteration of $F_1(r)=r-(r^THr/r^TH^2r)Hr$, where $H=AA^T$. If $A$ is symmetric PSD, $H$ can be taken as $A$. Next, for each $t=1, \dots, m$, we derive $F_t(r)=r- \sum_{i=1}^t \alpha_{t,i}(r) H^i r$ whose iterations correspond to a Krylov subspace method with restart. If $\kappa^+(H)$ is the ratio of the largest to smallest positive eigenvalues of $H$, when $Ax=b$ is consistent, in $k=O({\kappa^+(H)}{t^{-1}} \ln \varepsilon^{-1})$ iterations of $F_t$, $\Vert r_k \Vert \leq \varepsilon$. Each iteration takes $O(tN+t^3)$ operations, $N$ the number of nonzero entries in $A$. By directly applying $F_t$ to the normal equation, we get $\Vert A^TAx_k - A^Tb \Vert \leq \varepsilon$ in $O({\kappa^+(AA^T)}{t}^{-1} \ln \varepsilon^{-1})$ iterations. On the other hand, given any residual $r$, we compute $s$, the degree of its minimal polynomial with respect to $H$ in $O(sN+s^3)$ operations. Then $F_s(r)$ gives the minimum-norm solution of $Ax=b$ or an exact solution of $A^TAx=A^Tb$. The proposed algorithms are simple to implementation and theoretically robust. We present sample computational results, comparing the performance of CTA with CG and GMRES methods. The results support CTA as a highly competitive option.

Bounded perturbations resilient iterative methods for linear systems and least squares problems: operator-based approaches, analysis, and performance evaluation

On inertial Levenberg-Marquardt type methods for solving nonlinear ill-posed operator equations

Iterative Methods in GPU-Resident Linear Solvers for Nonlinear Constrained Optimization

A modified iteratively regularized Landweber iteration method: Hölder stability and convergence rates

Convergence and quasi-optimal cost of adaptive algorithms for nonlinear operators including iterative linearization and algebraic solver

A cut-and-project perspective for linearized Bregman iterations

Analysis of Generalized Iteratively Regularized Landweber Iterations driven by data

Randomized Iterative Methods for Linear Systems

Krasnoselskii-Mann Iterations: Inertia, Perturbations and Approximation

Bouligand-Newton type methods for non-smooth ill-posed problems

Interpolation Conditions for Linear Operators and Applications to Performance Estimation Problems

Low-rank-modified Galerkin methods for the Lyapunov equation

A Family of Iteration Functions for General Linear Systems

Landweber Scheme for Compact Operator Equation in Hilbert Space and Its Applications

Iterative Methods for Sparse Symmetric Multilinear Systems

Easily implementable iterative methods for variational inequalities with nonlinear diffusion–convection operator and constraints to the gradient of solution

A spectral iterative algorithm for solving constrained optimal control problems with nonquadratic functional

A Bregman-Kaczmarz method for nonlinear systems of equations

Modified iterated Tikhonov methods for solving systems of nonlinear ill-posed equations

Nonconforming Finite Element Discretisation for Semilinear Problems with Trilinear Nonlinearity

Perturbed conjugate gradient method: controlling semi-convergence, enforcing non-negativity, and accelerating iterations in tomographic imaging