Abstract:Recent results in the literature indicate that a residual network (ResNet) composed of a single residual block outperforms linear predictors, in the sense that all local minima in its optimization landscape are at least as good as the best linear predictor. However, these results are limited to a single residual block (i.e., shallow ResNets), instead of the deep ResNets composed of multiple residual blocks. We take a step towards extending this result to deep ResNets. We start by two motivating examples. First, we show that there exist datasets for which all local minima of a fully-connected ReLU network are no better than the best linear predictor, whereas a ResNet has strictly better local minima. Second, we show that even at the global minimum, the representation obtained from the residual block outputs of a 2-block ResNet do not necessarily improve monotonically over subsequent blocks, which highlights a fundamental difficulty in analyzing deep ResNets. Our main theorem on deep ResNets shows under simple geometric conditions that, any critical point in the optimization landscape is either (i) at least as good as the best linear predictor; or (ii) the Hessian at this critical point has a strictly negative eigenvalue. Notably, our theorem shows that a chain of multiple skip-connections can improve the optimization landscape, whereas existing results study direct skip-connections to the last hidden layer or output layer. Finally, we complement our results by showing benign properties of the "near-identity regions" of deep ResNets, showing depth-independent upper bounds for the risk attained at critical points as well as the Rademacher complexity.

Empirical Study on the Effect of Residual Networks on the Expressiveness of Linear Regions

Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks

Residual Recurrent Neural Networks for Learning Sequential Representations.

On the Number of Linear Regions of Convolutional Neural Networks

The Expressive Power of Neural Networks: A View from the Width

Are deep ResNets provably better than linear predictors?

Analysis on the Number of Linear Regions of Piecewise Linear Neural Networks

Deep Residual Learning for Nonlinear Regression

Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training

On the Number of Linear Regions of Convolutional Neural Networks with Piecewise Linear Activations.

Residual Networks of Residual Networks: Multilevel Residual Networks

On the Expressive Power of Neural Networks

Multi-level Residual Networks from Dynamical Systems View

Locally linear attributes of ReLU neural networks

A theoretical framework for deep locally connected ReLU network

Neural networks with ReLU powers need less depth

Residual Networks Behave Like Ensembles of Relatively Shallow Networks

Beyond Counting Linear Regions of Neural Networks, Simple Linear Regions Dominate!

Why ResNet Works? Residuals Generalize

Deep Residual Networks with Exponential Linear Unit