Abstract:Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. However, principled methods for measuring and manipulating iterative convergence in neural networks remain lacking. Here we address this gap by 1) quantifying the degree to which ResNets learn iterative solutions and 2) introducing a regularization approach that encourages the learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer-vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a “recurrent” ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively. To showcase the practicality of our approach, we study how iterative convergence impacts generalization on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or challenging recognition tasks with partial occlusions (Digitclutter). We find that iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets. Importantly, our approach may be useful for investigating other network architectures and tasks as well and we hope that our study provides a useful starting point for investigating the broader question of whether iterative convergence can help neural networks in their generalization.

Improving Object Detection With Convolutional Neural Network Via Iterative Mechanism

Improving object detection with deep convolutional networks via Bayesian optimization and structured prediction

Joint Feature-Level And Pixel-Level Domain Adaption For Object Detection In The Wild

Object Detection Based on Faster R-CNN Algorithm with Skip Pooling and Fusion of Contextual Information

Can neural networks benefit from objectives that encourage iterative convergent computations? A case study of ResNets and object classification

Conscious Inference for Object Detection

SELECTIVE CONVOLUTIONAL NETWORK: AN EFFICIENT OBJECT DETECTOR WITH IGNORING BACKGROUND

Improving Object Detection Via Improving Accuracy Of Object Localization

RD-IOD: Two-Level Residual-Distillation-Based Triple-Network for Incremental Object Detection

Faster ILOD: Incremental learning for object detectors based on faster RCNN

A Small Object Detection Algorithm Based on Improved Faster RCNN

Object Detection of Optical Remote Sensing Image Based on Improved Faster RCNN

Long-time Object Tracking Based on Hierarchical Convolution Features

Revisiting Class-Incremental Object Detection: an Efficient Approach Via Intrinsic Characteristics Alignment and Task Decoupling

Improving Object Detection Accuracy With Region And Regression Based Deep Cnns

Contrastive R-CNN for Incremental Learning in Object Detection.

Boosting Convolutional Features for Robust Object Proposals

An Improved Faster R-CNN for Small Object Detection

Improved Object Detection Algorithm Based on Faster RCNN

Improve Object Detection Via a Multi-Feature and Multi-Task CNN Model

Improved Object Detection Method of Micro-operating System Based on Region Convolutional Neural Network