From Fourier to Neural ODEs: Flow Matching for Modeling Complex Systems

Xin Li,Jingdong Zhang,Qunxi Zhu,Chengli Zhao,Xue Zhang,Xiaojun Duan,Wei Lin

2024-05-23

Abstract:Modeling complex systems using standard neural ordinary differential equations (NODEs) often faces some essential challenges, including high computational costs and susceptibility to local optima. To address these challenges, we propose a simulation-free framework, called Fourier NODEs (FNODEs), that effectively trains NODEs by directly matching the target vector field based on Fourier analysis. Specifically, we employ the Fourier analysis to estimate temporal and potential high-order spatial gradients from noisy observational data. We then incorporate the estimated spatial gradients as additional inputs to a neural network. Furthermore, we utilize the estimated temporal gradient as the optimization objective for the output of the neural network. Later, the trained neural network generates more data points through an ODE solver without participating in the computational graph, facilitating more accurate estimations of gradients based on Fourier analysis. These two steps form a positive feedback loop, enabling accurate dynamics modeling in our framework. Consequently, our approach outperforms state-of-the-art methods in terms of training time, dynamics prediction, and robustness. Finally, we demonstrate the superior performance of our framework using a number of representative complex systems.

Machine Learning,Physics Education

What problem does this paper attempt to address?

The problems that this paper attempts to solve are: when using standard Neural Ordinary Differential Equations (NODEs) to model complex systems, there are challenges such as high computational cost and being prone to getting trapped in local optimal solutions. Specifically, these challenges include that during the training process, NODEs may exhibit complex dynamic behaviors, such as violent fluctuations or stiffness problems, which significantly increase the time for numerical solution and the memory consumption of back - propagation. In addition, the adjoint method used to calculate gradients is also numerically sensitive.

To solve the above problems, the author proposes a framework named Fourier Neural Ordinary Differential Equations (FNODEs). This framework effectively trains NODEs by directly matching the target vector field estimated based on Fourier analysis without the need for simulation. The specific implementation methods are as follows:

1. **Utilize Fourier analysis theory**: Accurately estimate the gradient flow of the dynamic system from finite and noisy data, providing theoretical guarantees.
2. **Directly match the estimated gradient flow**: As a simulation - free framework, FNODEs can reduce the training time by more than ten times compared to standard NODEs.
3. **Data - enhanced training strategy**: Generate more data points to better estimate the gradient flow, and then retrain the model, forming a positive - feedback loop, which improves the robustness and accuracy of dynamic modeling.
4. **Resolution - invariant PDE modeling**: By incorporating the higher - order spatial gradients estimated based on Fourier analysis into the model, FNODEs can be easily applied to resolution - invariant modeling of PDEs, expanding the application range of classical NODEs.

Through these methods, the FNODEs framework not only improves the training speed and the robustness of the model, but also outperforms the existing state - of - the - art methods on multiple representative complex systems.