Abstract:Recent results in the literature indicate that a residual network (ResNet) composed of a single residual block outperforms linear predictors, in the sense that all local minima in its optimization landscape are at least as good as the best linear predictor. However, these results are limited to a single residual block (i.e., shallow ResNets), instead of the deep ResNets composed of multiple residual blocks. We take a step towards extending this result to deep ResNets. We start by two motivating examples. First, we show that there exist datasets for which all local minima of a fully-connected ReLU network are no better than the best linear predictor, whereas a ResNet has strictly better local minima. Second, we show that even at the global minimum, the representation obtained from the residual block outputs of a 2-block ResNet do not necessarily improve monotonically over subsequent blocks, which highlights a fundamental difficulty in analyzing deep ResNets. Our main theorem on deep ResNets shows under simple geometric conditions that, any critical point in the optimization landscape is either (i) at least as good as the best linear predictor; or (ii) the Hessian at this critical point has a strictly negative eigenvalue. Notably, our theorem shows that a chain of multiple skip-connections can improve the optimization landscape, whereas existing results study direct skip-connections to the last hidden layer or output layer. Finally, we complement our results by showing benign properties of the "near-identity regions" of deep ResNets, showing depth-independent upper bounds for the risk attained at critical points as well as the Rademacher complexity.

Convergence Analysis of Deep Residual Networks

Advances in Convolutional Neural Networks

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer

Convergence of Deep Neural Networks with General Activation Functions and Pooling

Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem

Deep Limits of Residual Neural Networks

Weighted Residuals for Very Deep Networks

Convergence Analysis for Over-Parameterized Deep Learning

Global Convergence of Gradient Descent for Deep Linear Residual Networks.

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths

Are deep ResNets provably better than linear predictors?

On Non-local Convergence Analysis of Deep Linear Networks.

Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks

Convergence Rate Analysis for Deep Ritz Method

Why ResNet Works? Residuals Generalize

An Overview of Convergence Analysis of Deep Neural Networks and the Visual Explanations of their Predictions

Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

On the Convergence of Deep Networks with Sample Quadratic Overparameterization