Abstract:Nonlinear operators with long-distance spatiotemporal dependencies are fundamental in modelling complex systems across sciences; yet, learning these non-local operators remains challenging in machine learning. Integral equations, which model such non-local systems, have wide-ranging applications in physics, chemistry, biology and engineering. We introduce the neural integral equation, a method for learning unknown integral operators from data using an integral equation solver. To improve scalability and model capacity, we also present the attentional neural integral equation, which replaces the integral with self-attention. Both models are grounded in the theory of second-kind integral equations, where the indeterminate appears both inside and outside the integral operator. We provide a theoretical analysis showing how self-attention can approximate integral operators under mild regularity assumptions, further deepening previously reported connections between transformers and integration, as well as deriving corresponding approximation results for integral operators. Through numerical benchmarks on synthetic and real-world data, including Lotka–Volterra, Navier–Stokes and Burgers' equations, as well as brain dynamics and integral equations, we showcase the models' capabilities and their ability to derive interpretable dynamics embeddings. Our experiments demonstrate that attentional neural integral equations outperform existing methods, especially for longer time intervals and higher-dimensional problems. Our work addresses a critical gap in machine learning for non-local operators and offers a powerful tool for studying unknown complex systems with long-range dependencies.

Nonlinear integro-differential operator regression with neural networks

Learning nonlinear integral operators via Recurrent Neural Networks and its application in solving Integro-Differential Equations

Long-time Integration of Nonlinear Wave Equations with Neural Operators

An application of artificial neural networks for solving fractional higher-order linear integro-differential equations

Learning integral operators via neural integral equations

Inversion of Integral Models: a Neural Network Approach

Neural Operators with Localized Integral and Differential Kernels

Nonlinear Reconstruction for Operator Learning of PDEs with Discontinuities

Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations

Neural Operator: Graph Kernel Network for Partial Differential Equations

Learning truly monotone operators with applications to nonlinear inverse problems

Deep learning schemes for parabolic nonlocal integro-differential equations

An Operator Learning Approach via Function-valued Reproducing Kernel Hilbert Space for Differential Equations

Neural Operator: Learning Maps Between Function Spaces

Deep learning architectures for nonlinear operator functions and nonlinear inverse problems

The Neural Network Approach to Inverse Problems in Differential Equations

Linearization Turns Neural Operators into Function-Valued Gaussian Processes

Nonparametric learning of kernels in nonlocal operators

Neural Operator Learning for Long-Time Integration in Dynamical Systems with Recurrent Neural Networks

Spectral methods for Neural Integral Equations

Neural Dynamical Operator: Continuous Spatial-Temporal Model with Gradient-Based and Derivative-Free Optimization Methods