Abstract:Recursive least squares (RLS) algorithms were once widely used for training small-scale neural networks, due to their fast convergence. However, previous RLS algorithms are unsuitable for training deep neural networks (DNNs), since they have high computational complexity and too many preconditions. In this paper, to overcome these drawbacks, we propose three novel RLS optimization algorithms for training feedforward neural networks, convolutional neural networks and recurrent neural networks (including long short-term memory networks), by using the error backpropagation and our average-approximation RLS method, together with the equivalent gradients of the linear least squares loss function with respect to the linear outputs of hidden layers. Compared with previous RLS optimization algorithms, our algorithms are simple and elegant. They can be viewed as an improved stochastic gradient descent (SGD) algorithm, which uses the inverse autocorrelation matrix of each layer as the adaptive learning rate. Their time and space complexities are only several times those of SGD. They only require the loss function to be the mean squared error and the activation function of the output layer to be invertible. In fact, our algorithms can be also used in combination with other first-order optimization algorithms without requiring these two preconditions. In addition, we present two improved methods for our algorithms. Finally, we demonstrate their effectiveness compared to the Adam algorithm on MNIST, CIFAR-10 and IMDB datasets, and investigate the influences of their hyperparameters experimentally.

Orthogonal Least Squares Algorithm for Training Cascade Neural Networks

Incremental Extreme Learning Machine Based On Cascade Neural Networks

Optimally Training a Cascade Classifier

Training Effective Node Classifiers for Cascade Classification

Harnessing Orthogonality to Train Low-Rank Neural Networks

IDK Cascades: Fast Deep Learning by Learning not to Overthink

Least Squares Training of Quadratic Convolutional Neural Networks with Applications to System Theory

Classical and Quantum Algorithms for Orthogonal Neural Networks

A Practical Layer-Parallel Training Algorithm for Residual Networks

Linear Asymmetric Classifier for cascade detectors

QuadraNet V2: Efficient and Sustainable Training of High-Order Neural Networks with Quadratic Adaptation

Efficient Vectorized Backpropagation Algorithms for Training Feedforward Networks Composed of Quadratic Neurons

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

Orthogonal least squares learning algorithm for radial basis function networks

Revisiting Recursive Least Squares for Training Deep Neural Networks

Block-diagonal Hessian-free Optimization for Training Neural Networks

Asymmetric Pruning for Learning Cascade Detectors

A Two-Stage Training Method for Modeling Constrained Systems With Neural Networks

A Structure-Guided Gauss-Newton Method for Shallow ReLU Neural Network

Joint Training of Cascaded Cnn for Face Detection

Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient