Predictions Based on Pixel Data: Insights from PDEs and Finite Differences

Elena Celledoni,James Jackaman,Davide Murari,Brynjulf Owren

2024-06-21

Abstract:As supported by abundant experimental evidence, neural networks are state-of-the-art for many approximation tasks in high-dimensional spaces. Still, there is a lack of a rigorous theoretical understanding of what they can approximate, at which cost, and at which accuracy. One network architecture of practical use, especially for approximation tasks involving images, is (residual) convolutional networks. However, due to the locality of the linear operators involved in these networks, their analysis is more complicated than that of fully connected neural networks. This paper deals with approximation of time sequences where each observation is a matrix. We show that with relatively small networks, we can represent exactly a class of numerical discretizations of PDEs based on the method of lines. We constructively derive these results by exploiting the connections between discrete convolution and finite difference operators. Our network architecture is inspired by those typically adopted in the approximation of time sequences. We support our theoretical results with numerical experiments simulating the linear advection, heat, and Fisher equations.

Numerical Analysis,Machine Learning

What problem does this paper attempt to address?

The paper mainly discusses how to use a two-layer convolutional neural network (CNNs) to accurately approximate the temporal discretization of partial differential equations (PDEs) based on the method of lines. The research indicates that by utilizing the connection between discrete convolution and finite difference operations, a relatively small network can accurately represent a class of PDE numerical discretizations. The paper not only constructs these results but also supports theoretical analysis through numerical experiments simulating linear transport, heat diffusion, and Fisher equations. The main objective proposed in the paper is to understand to what extent a two-layer CNN can accurately approximate the spatial-temporal discretization of PDEs. In the research, the authors view the temporal sequences as matrix sequences generated by the discretization of PDEs and focus on the two-dimensional spatial domain. They demonstrate that for linear PDEs, a two-layer CNN with ReLU activation function and two channels can provide second-order accuracy in semi-discretization. Similar results also apply to nonlinear PDEs with quadratic interaction terms. To improve prediction stability, the paper proposes two strategies: injecting noise during model training and preserving certain characteristics of the PDEs (such as gauge invariance) in the network, for example, preserving the norm of the initial condition when dealing with linear transport equations. The experiments show that these methods can improve the reliability of network predictions for the future. In summary, this paper aims to enhance the accuracy and stability of numerical solutions to PDEs through deep learning techniques, providing new tools for understanding and simulating complex physical processes.

Predictions Based on Pixel Data: Insights from PDEs and Finite Differences

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

Connections Between Numerical Algorithms for PDEs and Neural Networks

Finite Difference Neural Networks: Fast Prediction of Partial Differential Equations

Deep Neural Networks Motivated by Partial Differential Equations

Space-time deep neural network approximations for high-dimensional partial differential equations

Approximating High-Dimensional Minimal Surfaces with Physics-Informed Neural Networks

Functional SDE approximation inspired by a deep operator network architecture

Learning Time-Dependent PDEs with a Linear and Nonlinear Separate Convolutional Neural Network

Solutions to Elliptic and Parabolic Problems via Finite Difference Based Unsupervised Small Linear Convolutional Neural Networks

Solving partial differential equations with sampled neural networks

Neural networks catching up with finite differences in solving partial differential equations in higher dimensions

PDE-Net: Learning PDEs from Data

An Extreme Learning Machine-Based Method for Computational PDEs in Higher Dimensions

Deep neural network approximations for Monte Carlo algorithms

Translating Numerical Concepts for PDEs into Neural Architectures

Learning quantities of interest from parametric PDEs: An efficient neural-weighted Minimal Residual approach

Approximation of Solution Operators for High-dimensional PDEs

NeuralPDE: Modelling Dynamical Systems from Data

Physics-informed deep learning and compressive collocation for high-dimensional diffusion-reaction equations: practical existence theory and numerics

Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-order Backward Stochastic Differential Equations