Abstract:Recursive least squares (RLS) algorithms were once widely used for training small-scale neural networks, due to their fast convergence. However, previous RLS algorithms are unsuitable for training deep neural networks (DNNs), since they have high computational complexity and too many preconditions. In this paper, to overcome these drawbacks, we propose three novel RLS optimization algorithms for training feedforward neural networks, convolutional neural networks and recurrent neural networks (including long short-term memory networks), by using the error backpropagation and our average-approximation RLS method, together with the equivalent gradients of the linear least squares loss function with respect to the linear outputs of hidden layers. Compared with previous RLS optimization algorithms, our algorithms are simple and elegant. They can be viewed as an improved stochastic gradient descent (SGD) algorithm, which uses the inverse autocorrelation matrix of each layer as the adaptive learning rate. Their time and space complexities are only several times those of SGD. They only require the loss function to be the mean squared error and the activation function of the output layer to be invertible. In fact, our algorithms can be also used in combination with other first-order optimization algorithms without requiring these two preconditions. In addition, we present two improved methods for our algorithms. Finally, we demonstrate their effectiveness compared to the Adam algorithm on MNIST, CIFAR-10 and IMDB datasets, and investigate the influences of their hyperparameters experimentally.

Exactly Initialized Recursive Least Squares

Recursive Least Squares Support Vector Regression

CONVERGENCE AND STABILITY OF ITERATIVELY REWEIGHTED LEAST SQUARES FOR LOW-RANK MATRIXRECOVERY

A Sherman--Morrison--Woodbury approach to solving least squares problems with low-rank updates

Efficient Batch and Recursive Least Squares for Matrix Parameter Estimation

A new interpretation of the weighted pseudoinverse and its applications

Recursive Least-Squares Method with Membership Functions

Flexible Modified LSMR Method for Least Squares Problems

The Convergence of Least-Squares Progressive Iterative Approximation for Singular Least-Squares Fitting System.

On an Iterative Method for Solving the Least Squares Problem of Rank-Deficient Systems

Revisiting Recursive Least Squares for Training Deep Neural Networks

Recursive McCormick Linearization of Multilinear Programs

Good and Fast Row-Sparse ah-Symmetric Reflexive Generalized Inverses

Least Squares Properties of Generalized Inverses

Iterative algorithm for minimal norm least squares solution to general linear matrix equations

Generalized Forgetting Recursive Least Squares: Stability and Robustness Guarantees

Some Properties of Lsqr for Large Sparse Linear Least Squares Problems

Using $LDL^{T}$ factorizations in Newton's method for solving general large-scale algebraic Riccati equations

Mixed precision iterative refinement for least squares with linear equality constraints and generalized least squares problems

Alternately linearized implicit iteration methods for the minimal nonnegative solutions of the nonsymmetric algebraic Riccati equations

Smoothed Low Rank and Sparse Matrix Recovery by Iteratively Reweighted Least Squares Minimization