Abstract:In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress ($>50$\%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

A Bidirectional Feedforward Neural Network Architecture Using the Discretized Neural Memory Ordinary Differential Equation

On-chip 4F-System Based on Concave Mirrors for Optical Neural Networks

A Forward Learning Algorithm for Neural Memory Ordinary Differential Equations

HCFNN: High-order Coverage Function Neural Network for Image Classification

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

Towards bi-directional skip connections in encoder-decoder architectures and beyond

BiO-Net: Learning Recurrent Bi-directional Connections for Encoder-Decoder Architecture

Advancing Spiking Neural Networks Toward Deep Residual Learning

RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network

Convolution in Convolution for Network in Network

Nonrecurrent Neural Structure for Long-Term Dependence.

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Dynamic Multi-path Neural Network

Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency.

TFDMNet: A Novel Network Structure Combines the Time Domain and Frequency Domain Features

Multilayer Optoelectronic Hybrid Convolutional Neural Network with an Optical 4F-System Recurrent Structure

Deep Pulse-Coupled Neural Networks

Residual Feature-Reutilization Inception Network

A multi-path adaptive fusion network for multimodal brain tumor segmentation

Efficient Deep Spiking Multilayer Perceptrons With Multiplication-Free Inference