A weighted randomized Kaczmarz method for solving linear systems

Stefan Steinerberger

DOI: https://doi.org/10.1090/mcom/3644

IF: 2.118

2021-06-01

Mathematics of Computation

Abstract:Stefan Steinerberger Math. Comp. 90 (), 2815-2826. , references and article information

mathematics, applied

What problem does this paper attempt to address?

A Weighted Randomized Sparse Kaczmarz Method for Solving Linear Systems

Lu Zhang,Ziyang Yuan,Hongxia Wang,Hui Zhang

DOI: https://doi.org/10.1007/s40314-022-02105-9

2023-06-12

Abstract:The randomized sparse Kaczmarz method, designed for seeking the sparse solutions of the linear systems $Ax=b$, selects the $i$-th projection hyperplane with likelihood proportional to $\|a_{i}\|_2^2$, where $a_{i}^T$ is $i$-th row of $A$. In this work, we propose a weighted randomized sparse Kaczmarz method, which selects the $i$-th projection hyperplane with probability proportional to $\lvert\langle a_{i},x_{k}\rangle-b_{i}\rvert^p$, where $0<p<\infty$, for possible acceleration. It bridges the randomized Kaczmarz and greedy Kaczmarz by parameter $p$. Theoretically, we show its linear convergence rate in expectation with respect to the Bregman distance in the noiseless and noisy cases, which is at least as efficient as the randomized sparse Kaczmarz method. The superiority of the proposed method is demonstrated via a group of numerical experiments.

Numerical Analysis,Optimization and Control
A Deterministic Kaczmarz Algorithm for Solving Linear Systems

Changpeng Shao

DOI: https://doi.org/10.1137/21M1463306

IF: 1.908

2023-03-11

SIAM Journal on Matrix Analysis and Applications

Abstract:SIAM Journal on Matrix Analysis and Applications, Volume 44, Issue 1, Page 212-239, March 2023. We propose a new deterministic Kaczmarz algorithm for solving consistent linear systems [math]. Basically, the algorithm replaces orthogonal projections with reflections in the original scheme of Stefan Kaczmarz. Building on this, we give a geometric description of solutions of linear systems. Suppose [math] is [math], we show that the algorithm generates a series of points distributed with patterns on an [math]-sphere centered on a solution. These points lie evenly on [math] lower-dimensional spheres [math], with the property that for any [math], the midpoint of the centers of [math] is exactly a solution of [math]. With this discovery, we prove that taking the average of [math] points on any [math] effectively approximates a solution up to relative error [math], where [math] characterizes the eigengap of the orthogonal matrix produced by the product of [math] reflections generated by the rows of [math]. We also analyze the connection between [math] and [math], the condition number of [math]. In the worst case [math], while for random matrices [math] on average. Finally, we prove that the algorithm indeed solves the linear system [math], where [math] is the lower-triangular matrix such that [math]. The connection between this linear system and the original one is studied. The numerical tests indicate that this new Kaczmarz algorithm has comparable performance to randomized (block) Kaczmarz algorithms.

mathematics, applied
On the Randomized Kaczmarz Algorithm

Liang Dai,Mojtaba Soltanalian,Kristiaan Pelckmans

DOI: https://doi.org/10.1109/LSP.2013.2294376

2014-02-13

Abstract:The Randomized Kaczmarz Algorithm is a randomized method which aims at solving a consistent system of over determined linear equations. This note discusses how to find an optimized randomization scheme for this algorithm, which is related to the question raised by \cite{c2}. Illustrative experiments are conducted to support the findings.

Systems and Control,Optimization and Control
Randomized Extended Kaczmarz for Solving Least-Squares

Anastasios Zouzias,Nikolaos Freris

DOI: https://doi.org/10.1137/120889897

2013-01-06

Abstract:We present a randomized iterative algorithm that exponentially converges in expectation to the minimum Euclidean norm least squares solution of a given linear system of equations. The expected number of arithmetic operations required to obtain an estimate of given accuracy is proportional to the square condition number of the system multiplied by the number of non-zeros entries of the input matrix. The proposed algorithm is an extension of the randomized Kaczmarz method that was analyzed by Strohmer and Vershynin.

Numerical Analysis,Data Structures and Algorithms
Randomized Kaczmarz algorithm with averaging and block projection

Zeyi Zhang,Dong Shen

DOI: https://doi.org/10.1007/s10543-023-01002-9

IF: 1.856

2023-12-13

BIT Numerical Mathematics

Abstract:The randomized Kaczmarz algorithm is a simple iterative method for solving linear systems of equations. This study proposes a variant of the randomized Kaczmarz algorithm by combining block projection and weighted averaging techniques. Here, block projection quickly decreases iterative errors, and averaging reduces randomness and enables parallel computation simultaneously. Their combination can balance the convergence rate, convergence horizon, and computational complexity. In addition, three adaptive weights are designed to balance multiple block calculations and accelerate the proposed method. Exponential convergence is established for general linear systems (overdetermined or underdetermined, full-rank or deficient-rank, and consistent or inconsistent). Numerical simulations explain and verify the results.

mathematics, applied,computer science, software engineering
Randomized Kaczmarz for tensor linear systems

Anna Ma,Denali Molitor

DOI: https://doi.org/10.1007/s10543-021-00877-w

IF: 1.856

2021-05-17

BIT Numerical Mathematics

Abstract:Solving linear systems of equations is a fundamental problem in mathematics. When the linear system is so large that it cannot be loaded into memory at once, iterative methods such as the randomized Kaczmarz method excel. Here, we extend the randomized Kaczmarz method to solve multi-linear (tensor) systems under the tensor–tensor t-product. We present convergence guarantees for tensor randomized Kaczmarz in two ways: using the classical matrix randomized Kaczmarz analysis and taking advantage of the tensor–tensor t-product structure. We demonstrate experimentally that the tensor randomized Kaczmarz method converges faster than traditional randomized Kaczmarz applied to a naively matricized version of the linear system. In addition, we draw connections between the proposed algorithm and a previously known extension of the randomized Kaczmarz algorithm for matrix linear systems.

mathematics, applied,computer science, software engineering
An Accelerated Block Randomized Kaczmarz Method

Anjie Qiao,Xuezhong Wang,Yimin Wei

DOI: https://doi.org/10.1007/978-981-99-2310-6_7

2023-01-01

Abstract:The Kaczmarz method is a kind of row iterative method for solving large-scale linear equations. In this paper, we give a block accelerated randomized Kaczmarz method which is the block version of the accelerated randomized Kaczmarz method to deal with real linear systems. We analyze the upper bounds of the mean squared error and the estimate of the convergence rate of the algorithm. Moreover, we use the method to solve the real system of t-product tensor equations which has been paid a great deal of attention in recent years. Several numerical examples are presented to illustrate our methods.
Randomized block Kaczmarz method with projection for solving least squares

Deanna Needell,Ran Zhao,Anastasios Zouzias

DOI: https://doi.org/10.1016/j.laa.2015.06.027

IF: 1.307

2015-11-01

Linear Algebra and its Applications

Abstract:The Kaczmarz method is an iterative method for solving overcomplete linear systems of equations Ax=b. The randomized version of the Kaczmarz method put forth by Strohmer and Vershynin iteratively projects onto a randomly chosen solution space given by a single row of the matrix A and converges linearly1 in expectation to the solution of a consistent system. In this paper we analyze two block versions of the method each with a randomized projection, designed to converge in expectation to the least squares solution, often faster than the standard variants. Our approach utilizes both a row and column-paving of the matrix A to guarantee linear convergence when the matrix has consistent row norms (called nearly standardized), and a single column-paving when the row norms are unbounded. The proposed methods are an extension of the block Kaczmarz method analyzed by Needell and Tropp and the Randomized Extended Kaczmarz method of Zouzias and Freris. The contribution is thus two-fold; unlike the standard Kaczmarz method, our results demonstrate convergence to the least squares solution of inconsistent systems (both methods in the nearly standardized case and the second method in other cases). By using appropriate blocks of the matrix this convergence can be significantly accelerated, as is demonstrated by numerical experiments.

mathematics, applied
Randomized Kaczmarz Converges Along Small Singular Vectors

Stefan Steinerberger

DOI: https://doi.org/10.1137/20M1350947

IF: 1.908

2021-04-09

SIAM Journal on Matrix Analysis and Applications

Abstract:SIAM Journal on Matrix Analysis and Applications, Volume 42, Issue 2 , Page 608-615, January 2021. Randomized Kaczmarz is a simple iterative method for finding solutions of linear systems . We point out that the arising sequence tends to converge to the solution in an interesting way: generically, as , tends to the singular vector of corresponding to the smallest singular value. This has interesting consequences: in particular, the error analysis of Strohmer and Vershynin is optimal. It also quantifies the "preconvergence" phenomenon where the method initially seems to converge faster. This fact also allows for a fast computation of vectors for which the Rayleigh quotient is small: solve via randomized Kaczmarz.

mathematics, applied
A Kaczmarz Method with Simple Random Sampling for Solving Large Linear Systems

Yutong Jiang,Gang Wu,Long Jiang

DOI: https://doi.org/10.48550/arXiv.2011.14693

2020-11-30

Numerical Analysis

Abstract:The Kaczmarz method is a popular iterative scheme for solving large-scale linear systems. The randomized Kaczmarz method (RK) greatly improves the convergence rate of the Kaczmarz method, by using the rows of the coefficient matrix in random order rather than in their given order. An obvious disadvantage of the randomized Kaczmarz method is its probability criterion for selecting the active or working rows in the coefficient matrix. In [{\sc Z.Z. Bai, W. Wu}, {\em On greedy randomized Kaczmarz method for solving large sparse linear systems}, SIAM Journal on Scientific Computing, 2018, 40: A592--A606], the authors proposed a greedy randomized Kaczmarz method (GRK). However, this method may suffer from heavily computational cost when the size of the matrix is large, and the overhead will be prohibitively large for big data problems. The contribution of this work is as follows. First, from the probability significance point of view, we present a partially randomized Kaczmarz method, which can reduce the computational overhead needed in greedy randomized Kaczmarz method. Second, based on Chebyshev's law of large numbers and Z-test, we apply a simple sampling approach to the partially randomized Kaczmarz method. The convergence of the proposed method is established. Third, we apply the new strategy to the ridge regression problem, and propose a partially randomized Kaczmarz method with simple random sampling for ridge regression. Numerical experiments demonstrate the superiority of the new algorithms over many state-of-the-art randomized Kaczmarz methods for large linear systems problems and ridge regression problems.
Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

Yun Zeng,Deren Han,Yansheng Su,Jiaxin Xie

DOI: https://doi.org/10.1007/s11075-023-01540-x

IF: 2.37

2023-06-02

Numerical Algorithms

Abstract:We investigate the randomized Kaczmarz method that adaptively updates the stepsize using readily available information for solving inconsistent linear systems. A novel geometric interpretation is provided which shows that the proposed method can be viewed as an orthogonal projection method in some sense. We prove that this method converges linearly in expectation to the unique minimum Euclidean norm least-squares solution of the linear system, and provide a tight upper bound for the convergence of the proposed method. Numerical experiments are also given to illustrate the theoretical results.

mathematics, applied
Faster Randomized Block Kaczmarz Algorithms

Ion Necoara

DOI: https://doi.org/10.1137/19M1251643

IF: 1.908

2019-11-27

SIAM Journal on Matrix Analysis and Applications

Abstract:The Kaczmarz algorithm is a simple iterative scheme for solving consistent linear systems. At each step, the method projects the current iterate onto the solution space of a single constraint. Hence, it requires low cost per iteration and storage, and it has a linear rate of convergence. Distributed implementations of Kaczmarz have recently become the de facto architectural choice for large-scale linear systems. Therefore, in this paper we develop a family of randomized block Kaczmarz algorithms that uses at each step a subset of the constraints and extrapolated stepsizes, and can be deployed on distributed computing units. Our approach is based on several new ideas and tools, including stochastic selection rules for the blocks of rows, stochastic conditioning of linear systems, and novel strategies for designing extrapolated stepsizes. We prove that randomized block Kaczmarz algorithms converge linearly in expectation, with a rate depending on the geometric properties of the matrix and its submatrices and on the size of the blocks. Our convergence analysis reveals that the algorithm is most effective when it is given a good sampling of the rows into well-conditioned blocks. Besides providing a general framework for the design and analysis of randomized block Kaczmarz methods, our results resolve an open problem in the literature related to the theoretical understanding of observed practical efficiency of extrapolated block Kaczmarz methods. We also propose an accelerated block Kaczmarz scheme, that is, acceleration in the sense of Chebyshev semi-iterative methods, where the stepsize is chosen based on the roots of Chebyshev polynomials, and we derive convergence rates depending on the square root of the geometric properties of the matrix. Finally, numerical examples illustrate the benefits of the new algorithms.

mathematics, applied
Kaczmarz Kac Walk

Stefan Steinerberger

2024-11-11

Abstract:The Kaczmarz method is a way to iteratively solve a linear system of equations $Ax = b$. One interprets the solution $x$ as the point where hyperplanes intersect and then iteratively projects an approximate solution onto these hyperplanes to get better and better approximations. We note a somewhat related idea: one could take two random hyperplanes and project one into the orthogonal complement of the other. This leads to a sequence of linear systems $A^{(k)} x = b^{(k)}$ which is fast to compute, preserves the original solution and whose small singular values grow like $\sigma_{\ell}(A^{(k)}) \sim \exp(k/n^2) \cdot \sigma_{\ell}(A)$.

Numerical Analysis,Probability
Inertial randomized Kaczmarz algorithms for solving coherent linear systems

Songnian He,Ziting Wang,Qiao-Li Dong

2023-06-14

Abstract:In this paper, by regarding the two-subspace Kaczmarz method [20] as an alternated inertial randomized Kaczmarz algorithm we present a new convergence rate estimate which is shown to be better than that in [20] under a mild condition. Furthermore, we accelerate the alternated inertial randomized Kaczmarz algorithm and introduce a multi-step inertial randomized Kaczmarz algorithm which is proved to have a faster convergence rate. Numerical experiments support the theory results and illustrate that the multi-inertial randomized Kaczmarz algorithm significantly outperform the two-subspace Kaczmarz method in solving coherent linear systems.

Numerical Analysis,Optimization and Control
A semi-randomized Kaczmarz method with simple random sampling for large-scale linear systems

Jiang, Yutong,Jiang, Long

DOI: https://doi.org/10.1007/s10444-023-10018-2

2023-03-11

Advances in Computational Mathematics

Abstract:Randomized Kaczmarz-type methods are appealing for large-scale linear systems arising from big data problems. One of the keys of randomized Kaczmarz-type methods is how to effectively select working rows from the coefficient matrix. To the best of our knowledge, most of the randomized Kaczmarz-type methods need to compute probabilities for choosing working rows. However, when the amount of data is huge, the computation of probabilities will be inaccurate due to many factors such as rounding errors and data distribution. Moreover, in some popular randomized Kaczmarz methods, we have to scan all the rows of the data matrix in advance, or to compute residual of the linear system in each step. Hence, we have to access all the rows of the data matrix, which are unfavorable for big data problems. To overcome these difficulties, we first introduce a semi-randomized Kaczmarz method in which there is no need to compute probabilities explicitly. However, we still have to access all the rows of the matrix for the computation of residuals. To improve the semi-randomized Kaczmarz method further, inspired by Chebyshev's (weak) law of large numbers, we apply the simple sampling strategy to the semi-randomized Kaczmarz method, and propose a semi-randomized Kaczmarz method with simple random sampling. In the new method, there is no need to calculate probabilities explicitly and it is free of computing residuals of the linear system, and is free of constructing index sets via scanning residuals. Indeed, we only need to compute some elements of residuals corresponding to simple sampling sets, and a small portion of rows of the matrix are utilized. Convergence results are established to show the rationality and feasibility of the two proposed methods. Numerical experiments demonstrate the superiority of the new methods over many state-of-the-art randomized Kaczmarz methods for large-scale linear systems.

mathematics, applied
On Multi-Step Randomized Extended Kaczmarz Method for Solving Large Sparse Inconsistent Linear Systems

Zhong-Zhi Bai,Lu Wang

DOI: https://doi.org/10.1016/j.apnum.2023.06.008

IF: 2.994

2023-06-21

Applied Numerical Mathematics

Abstract:In order to improve the convergence property and computational behavior of the randomized extended Kaczmarz method, we propose a multi-step randomized extended Kaczmarz method, in which we repeatedly update the iterate several times at each iteration step, obtaining a nonstationary inner-outer iteration scheme for solving large-scale, sparse, and inconsistent system of linear equations. For this multi-step randomized extended Kaczmarz method, we prove its convergence, derive an upper bound for its convergence rate, and demonstrate that this upper bound can be smaller than that of the randomized extended Kaczmarz method for several typical choices of the numbers of inner iteration steps. Numerical experiments also show that the multi-step randomized extended Kaczmarz method can perform better than the randomized extended Kaczmarz method if we choose the numbers of inner iteration steps appropriately.

mathematics, applied
Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

Chuang Wang,Ameya Agaskar,Yue M. Lu

DOI: https://doi.org/10.48550/arXiv.1502.00190

2015-02-01

Abstract:We provide a complete characterization of the randomized Kaczmarz algorithm (RKA) for inconsistent linear systems. The Kaczmarz algorithm, known in some fields as the algebraic reconstruction technique, is a classical method for solving large-scale overdetermined linear systems through a sequence of projection operators; the randomized Kaczmarz algorithm is a recent proposal by Strohmer and Vershynin to randomize the sequence of projections in order to guarantee exponential convergence (in mean square) to the solutions. A flurry of work followed this development, with renewed interest in the algorithm, its extensions, and various bounds on their performance. Earlier, we studied the special case of consistent linear systems and provided an exact formula for the mean squared error (MSE) in the value reconstructed by RKA, as well as a simple way to compute the exact decay rate of the error. In this work, we consider the case of inconsistent linear systems, which is a more relevant scenario for most applications. First, by using a "lifting trick", we derive an exact formula for the MSE given a fixed noise vector added to the measurements. Then we show how to average over the noise when it is drawn from a distribution with known first and second-order statistics. Finally, we demonstrate the accuracy of our exact MSE formulas through numerical simulations, which also illustrate that previous upper bounds in the literature may be several orders of magnitude too high.

Numerical Analysis,Information Theory,Optimization and Control
Paved with good intentions: Analysis of a randomized block Kaczmarz method

Deanna Needell,Joel A. Tropp

DOI: https://doi.org/10.1016/j.laa.2012.12.022

IF: 1.307

2014-01-01

Linear Algebra and its Applications

Abstract:The block Kaczmarz method is an iterative scheme for solving overdetermined least-squares problems. At each step, the algorithm projects the current iterate onto the solution space of a subset of the constraints. This paper describes a block Kaczmarz algorithm that uses a randomized control scheme to choose the subset at each step. This algorithm is the first block Kaczmarz method with an (expected) linear rate of convergence that can be expressed in terms of the geometric properties of the matrix and its submatrices. The analysis reveals that the algorithm is most effective when it is given a good row paving of the matrix, a partition of the rows into well-conditioned blocks. The operator theory literature provides detailed information about the existence and construction of good row pavings. Together, these results yield an efficient block Kaczmarz scheme that applies to many overdetermined least-squares problem.

mathematics, applied
A Randomized Distributed Kaczmarz Algorithm and Anomaly Detection

Fritz Keinert,Eric S. Weber

DOI: https://doi.org/10.3390/axioms11030106

2022-02-26

Axioms

Abstract:The Kaczmarz algorithm is an iterative method for solving systems of linear equations. We introduce a randomized Kaczmarz algorithm for solving systems of linear equations in a distributed environment, i.e., the equations within the system are distributed over multiple nodes within a network. The modification we introduce is designed for a network with a tree structure that allows for passage of solution estimates between the nodes in the network. We demonstrate that the algorithm converges to the solution, or the solution of minimal norm, when the system is consistent. We also prove convergence rates of the randomized algorithm that depend on the spectral data of the coefficient matrix and the random control probability distribution. In addition, we demonstrate that the randomized algorithm can be used to identify anomalies in the system of equations when the measurements are perturbed by large, sparse noise.

A weighted randomized Kaczmarz method for solving linear systems

A Weighted Randomized Sparse Kaczmarz Method for Solving Linear Systems

A Deterministic Kaczmarz Algorithm for Solving Linear Systems

On the Randomized Kaczmarz Algorithm

Randomized Extended Kaczmarz for Solving Least-Squares

Randomized Kaczmarz algorithm with averaging and block projection

Randomized Kaczmarz for tensor linear systems

An Accelerated Block Randomized Kaczmarz Method

Randomized block Kaczmarz method with projection for solving least squares

Randomized Kaczmarz Converges Along Small Singular Vectors

A Kaczmarz Method with Simple Random Sampling for Solving Large Linear Systems

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

Faster Randomized Block Kaczmarz Algorithms

Kaczmarz Kac Walk

Inertial randomized Kaczmarz algorithms for solving coherent linear systems

A semi-randomized Kaczmarz method with simple random sampling for large-scale linear systems

On Multi-Step Randomized Extended Kaczmarz Method for Solving Large Sparse Inconsistent Linear Systems

Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

Paved with good intentions: Analysis of a randomized block Kaczmarz method

A Randomized Distributed Kaczmarz Algorithm and Anomaly Detection