Abstract:Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. However, principled methods for measuring and manipulating iterative convergence in neural networks remain lacking. Here we address this gap by 1) quantifying the degree to which ResNets learn iterative solutions and 2) introducing a regularization approach that encourages the learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer-vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a “recurrent” ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively. To showcase the practicality of our approach, we study how iterative convergence impacts generalization on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or challenging recognition tasks with partial occlusions (Digitclutter). We find that iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets. Importantly, our approach may be useful for investigating other network architectures and tasks as well and we hope that our study provides a useful starting point for investigating the broader question of whether iterative convergence can help neural networks in their generalization.

On the rate of convergence of perceptron learning

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

A Discussion of the Learning in a Simple Perceptron

On Accelerated Perceptrons and Beyond

On Convergence Rate of a Class of Genetic Algorithms

Generalization ability of a perceptron with non-monotonic transfer function

Learning in A Higher-Order Simple Perceptron

On the rates of convergence for learning with convolutional neural networks

Fast Convergence in Learning Two-Layer Neural Networks with Separable Data

Leveraging the two timescale regime to demonstrate convergence of neural networks

Neurons learn slower than they think

On the Convergence of Gradient Descent for Large Learning Rates

Perceptron Training Algorithms Designed Using Discrete-Time Control Liapunov Functions

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

On Convergence Rate of MRetrace

On the convergence of gradient descent for two layer neural networks

An Algorithm for Fast Convergence of Back Propagation by Enlarging Error

Can neural networks benefit from objectives that encourage iterative convergent computations? A case study of ResNets and object classification

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

On the Perceptron's Compression

A New Convergence Property of Online BP Learning