Abstract:Physics-informed neural networks (PINNs) and their variants have recently emerged as alternatives to traditional partial differential equation (PDE) solvers, but little literature has focused on devising accurate numerical integration methods for neural networks (NNs), which is essential for getting accurate solutions. In this work, we propose adaptive quadratures for the accurate integration of neural networks and apply them to loss functions appearing in low-dimensional PDE discretisations. We show that at opposite ends of the spectrum, continuous piecewise linear (CPWL) activation functions enable one to bound the integration error, while smooth activations ease the convergence of the optimisation problem. We strike a balance by considering a CPWL approximation of a smooth activation function. The CPWL activation is used to obtain an adaptive decomposition of the domain into regions where the network is almost linear, and we derive an adaptive global quadrature from this mesh. The loss function is then obtained by evaluating the smooth network (together with other quantities, e.g., the forcing term) at the quadrature points. We propose a method to approximate a class of smooth activations by CPWL functions and show that it has a quadratic convergence rate. We then derive an upper bound for the overall integration error of our proposed adaptive quadrature. The benefits of our quadrature are evaluated on a strong and weak formulation of the Poisson equation in dimensions one and two. Our numerical experiments suggest that compared to Monte-Carlo integration, our adaptive quadrature makes the convergence of NNs quicker and more robust to parameter initialisation while needing significantly fewer integration points and keeping similar training times.

Learning and approximation capabilities of adaptive spline activation function neural networks

Multilayer feedforward networks with adaptive spline activation function

A robust learning algorithm for feedforward neural networks with adaptive spline activation function

SplineGen: Approximating Unorganized Points Through Generative AI

A comparison of deep networks with ReLU activation function and linear spline-type methods

B-Spline Recurrent Neurall Network and Its Application to Modelling

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

An Adaptive Learning Rate for the Training of B-Spline Networks

Estimating Functions and Derivatives Via Adaptive Penalized Splines.

Adaptive optimization of isogeometric multi-patch discretizations using artificial neural networks

Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Adaptive quadratures for nonlinear approximation of low-dimensional PDEs using smooth neural networks

A Rapid Two-Step Learning Algorithm for Spline Activation Function Neural Networks with the Application on Biped Gait Recognition

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Consensus Function from an $L_p^q-$norm Regularization Term for its Use as Adaptive Activation Functions in Neural Networks

Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics

Nonparametric Automatic Differentiation Variational Inference with Spline Approximation

Efficient spline regression for neural spiking data

The Performance of Cross-Validation and Maximum Likelihood Estimators of Spline Smoothing Parameters