Abstract:The conjoining of dynamical systems and deep learning has become a topic of great interest. In particular, neural differential equations (NDEs) demonstrate that neural networks and differential equation are two sides of the same coin. Traditional parameterised differential equations are a special case. Many popular neural network architectures, such as residual networks and recurrent networks, are discretisations.
NDEs are suitable for tackling generative problems, dynamical systems, and time series (particularly in physics, finance, ...) and are thus of interest to both modern machine learning and traditional mathematical modelling. NDEs offer high-capacity function approximation, strong priors on model space, the ability to handle irregular data, memory efficiency, and a wealth of available theory on both sides.
This doctoral thesis provides an in-depth survey of the field.
Topics include: neural ordinary differential equations (e.g. for hybrid neural/mechanistic modelling of physical systems); neural controlled differential equations (e.g. for learning functions of irregular time series); and neural stochastic differential equations (e.g. to produce generative models capable of representing complex stochastic dynamics, or sampling from complex high-dimensional distributions).
Further topics include: numerical methods for NDEs (e.g. reversible differential equations solvers, backpropagation through differential equations, Brownian reconstruction); symbolic regression for dynamical systems (e.g. via regularised evolution); and deep implicit models (e.g. deep equilibrium models, differentiable optimisation).
We anticipate this thesis will be of interest to anyone interested in the marriage of deep learning with dynamical systems, and hope it will provide a useful reference for the current state of the art.
Machine Learning,Classical Analysis and ODEs,Dynamical Systems,Numerical Analysis
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to explore the application potential of neural differential equations (NDEs) in modern machine learning and traditional mathematical modeling. Specifically, the paper aims to:
1. **Review the field of neural differential equations**: Provide a comprehensive review covering different types of NDEs such as neural ordinary differential equations (NODEs), neural controlled differential equations (NCDEs), and neural stochastic differential equations (NSDEs).
2. **Explore the advantages of NDEs**: Discuss the advantages of NDEs in dealing with generation problems, dynamic systems, and time series (especially in fields such as physics and finance), including high - capacity function approximation, strong priors in the model space, the ability to handle irregular data, memory efficiency, and a rich theoretical foundation.
3. **Introduce specific applications of NDEs**:
- **Neural ordinary differential equations (NODEs)**: Applied to image classification, hybrid neural/mechanical modeling of physical systems, continuous normalization flows, latent ODEs, and residual networks, etc.
- **Neural controlled differential equations (NCDEs)**: Used for learning functions of irregular time series, handling long - time series, and rough differential equations, etc.
- **Neural stochastic differential equations (NSDEs)**: Generate models that can represent complex stochastic dynamics or sample from complex high - dimensional distributions.
4. **Discuss numerical methods and techniques**: Including reversible differential equation solvers, back - propagation through differential equations, Brownian motion reconstruction, etc., to improve the computational efficiency and accuracy of NDEs.
5. **Explore symbolic regression and deep hidden - layer models**: Introduce the application of symbolic regression in dynamic systems, and the concept and application of deep hidden - layer models (such as deep equilibrium models).
Through these contents, the paper hopes to provide a useful reference for readers interested in the combination of deep learning and dynamic systems, and show the latest progress in this field at present.