Abstract:In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress ($>50$\%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

A new approach to neural networks using pseudo-differential operators

Meta-learning pseudo-differential operators with deep neural networks

Pseudo-Differential Neural Operator: Generalized Fourier Neural Operator for Learning Solution Operators of Partial Differential Equations

Learning Partial Differential Equations with Deep Parallel Neural Operator

Neural Operator: Graph Kernel Network for Partial Differential Equations

Neural Operator: Learning Maps Between Function Spaces

Enhancing Solutions for Complex PDEs: Introducing Complementary Convolution and Equivariant Attention in Fourier Neural Operators

Convection-Diffusion Equation: A Theoretically Certified Framework for Neural Networks

The Neural Network Approach to Inverse Problems in Differential Equations

Pseudo-differential integral autoencoder network for inverse PDE operators

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Diffeomorphic Latent Neural Operators for Data-Efficient Learning of Solutions to Partial Differential Equations

Neural Operators with Localized Integral and Differential Kernels

Application of Nonlinear Fractional Differential Equations in Computer Artificial Intelligence Algorithms

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Functional SDE approximation inspired by a deep operator network architecture

PDE Models for Deep Neural Networks: Learning Theory, Calculus of Variations and Optimal Control

Convolutional Neural Operators for robust and accurate learning of PDEs

A Neural Network Training Method Based on Distributed PID Control

Computation of Pseudo-Differential Operators.