Abstract:When the CG method for solving linear algebraic systems was formulated about 70 years ago by Lanczos, Hestenes, and Stiefel, it was considered an iterative process possessing a mathematical finite termination property. CG was placed into a rich mathematical context, including links with Gauss quadrature and continued fractions. The optimality property of CG was described via a normalized weighted polynomial least squares approximation to zero. This highly nonlinear problem explains the adaptation of CG iterates to the given data. Karush and Hayes immediately considered CG in infinite dimensional Hilbert spaces and investigated its superlinear convergence. Since then, the view of CG and other Krylov subspace methods has changed. Today these methods are primarily used as computational tools, and their behavior is typically characterized using linear upper bounds or heuristics based on clustering of eigenvalues. Such simplifications limit the mathematical understanding and also negatively affect their practical application. This paper offers a different perspective. Focusing on CG and GMRES, it presents mathematically important and practically relevant phenomena that uncover their behavior through a discussion of computed examples. These examples provide an easily accessible approach that enables understanding of the methods, while pointers to more detailed analyses in the literature are given. This approach allows readers to choose the level of depth and thoroughness appropriate for their intentions. Some of the points made in this paper illustrate well known facts. Others challenge mainstream views and explain existing misunderstandings. Several points refer to recent results leading to open problems. We consider CG and GMRES crucially important for the mathematical understanding, further development, and practical applications also of other Krylov subspace methods.

Pseudoeigenvector bases and deflated GMRES for highly nonnormal matrices

New Convergence Analysis of GMRES with Weighted Norms, Preconditioning and Deflation, Leading to a New Deflation Space

Normalized Newton method to solve generalized tensor eigenvalue problems

Randomized flexible GMRES with deflated restarting

GMRES, pseudospectra, and Crouzeix’s conjecture for shifted and scaled Ginibre matrices

GMRES, pseudospectra, and Crouzeix's conjecture for shifted and scaled Ginibre matrices

Restarted GMRES Method Augmented with the Combination of Harmonic Ritz Vectors and Error Approximations

The Projection Methods for Computing the Pseudospectra of Large Scale Matrices

New Convergence Analysis of GMRES with Weighted Norms, Preconditioning, and Deflation, Leading to a New Deflation Space

Deflated Iterative Methods for Linear Equations with Multiple Right-Hand Sides

Global FOM and GMRES algorithms for a class of complex matrix equations.

On IGMRES: an Incomplete Generalized Minimal Residual Method for Large Unsymmetric Linear Systems

Growth Factors of Orthogonal Matrices and Local Behavior of Gaussian Elimination with Partial and Complete Pivoting

Block GMRES Method with Inexact Breakdowns and Deflated Restarting

Compressed Basis GMRES on High Performance GPUs

Theoretical and numerical comparisons of GMRES and WZ-GMRES

Generalized Eigenspaces and Pseudospectra of Nonnormal and Defective Matrix-Valued Dynamical Systems

Deflation of Eigenvalues for GMRES in Lattice QCD

Restarted Generalized Krylov Subspace Methods for Solving Large-Scale Polynomial Eigenvalue Problems

Weighted and deflated global GMRES algorithms for solving large Sylvester matrix equations

Towards understanding CG and GMRES through examples