Abstract:Neural ordinary differential equations (NODEs), one of the most influential works of the differential equation-based deep learning, are to continuously generalize residual networks and opened a new field. They are currently utilized for various downstream tasks, e.g., image classification, time series classification, image generation, etc. Its key part is how to model the time-derivative of the hidden state, denoted dh(t)/dt. People have habitually used conventional neural network architectures, e.g., fully-connected layers followed by non-linear activations. In this paper, however, we present a neural operator-based method to define the time-derivative term. Neural operators were initially proposed to model the differential operator of partial differential equations (PDEs). Since the time-derivative of NODEs can be understood as a special type of the differential operator, our proposed method, called branched Fourier neural operator (BFNO), makes sense. In our experiments with general downstream tasks, our method significantly outperforms existing methods.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of how to improve the modeling of the time - derivative term \(\frac{dh(t)}{dt}\) in Neural Ordinary Differential Equations (NODEs). Specifically, the author proposes a method based on neural operators to define this time - derivative term in order to improve the performance of NODEs in various downstream tasks. #### Main problems: 1. **Limitations of existing methods**: - At present, most NODEs use traditional neural network architectures (such as fully - connected layers and convolutional layers) to model the time - derivative term \(\frac{dh(t)}{dt}\). Although these methods are effective, their expressive ability on complex tasks is limited. - Neural operators were initially used to model differential operators of Partial Differential Equations (PDEs), and ODEs can be regarded as a special case of PDEs. Therefore, it is theoretically feasible and has potential to apply neural operators to the time - derivative term of ODEs. 2. **Improving model performance**: - The author hopes to improve the performance of NODEs in downstream tasks such as image classification, time - series classification and image generation by more accurate modeling of the time - derivative term. #### Solutions: - **Branched Fourier Neural Operator (BFNO)**: The author proposes a new architecture named BFNO, which learns more complex neural operators through dynamic global convolution operations and multi - branch structures, so as to better capture the changes of the time - derivative term. - **Experimental verification**: Through experiments on multiple benchmark datasets, it is proved that BFNO - NODE has higher performance compared with other existing methods. ### Formula representation: - The form of the ODE function is: \[ \frac{dh(t)}{dt}=f(h(t), t; \theta_f) \] where \(\theta_f\) represents the learnable parameters. - The update process of the BFNO layer can be represented as: \[ g_{k + 1}(x)=\sigma\left(F^{-1}(\rho(F(g_k)))(x)+Wg_k(x)\right) \] where: - \(F\) and \(F^{-1}\) represent the fast Fourier transform and its inverse transform respectively. - \(\rho\) is the dynamic global convolution operation. - \(W\) is the linear transformation matrix. - \(\sigma\) is the activation function. In this way, BFNO - NODE can significantly improve the expressive ability and performance of the model while maintaining computational efficiency.

Operator-learning-inspired Modeling of Neural Ordinary Differential Equations

Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics

LFT: Neural Ordinary Differential Equations With Learnable Final-Time

Extended dynamic mode decomposition with dictionary learning using neural ordinary differential equations

Learning Partial Differential Equations with Deep Parallel Neural Operator

HiDeS: a higher-order-derivative-supervised neural ordinary differential equation for multi-robot systems and opinion dynamics

Parameter-varying neural ordinary differential equations with partition-of-unity networks

Parameterized neural ordinary differential equations: applications to computational physics problems

Neural Delay Differential Equations: System Reconstruction and Image Classification

Pseudo-Differential Neural Operator: Generalized Fourier Neural Operator for Learning Solution Operators of Partial Differential Equations

Learning the Delay Using Neural Delay Differential Equations

The advance of neural ordinary differential ordinary differential equations

Neural Ordinary Differential Equations for Nonlinear System Identification

On Second Order Behaviour in Augmented Neural ODEs

Modulated Neural ODEs

Diffeomorphism Neural Operator for various domains and parameters of partial differential equations

Deep Neural Network Modeling of Unknown Partial Differential Equations in Nodal Space

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Derivative-enhanced Deep Operator Network

Balanced Neural ODEs: nonlinear model order reduction and Koopman operator approximations