Abstract:We consider the problem of one-dimensional function approximation using shallow neural networks (NN) with a rectified linear unit (ReLU) activation function and compare their training with traditional methods such as univariate Free Knot Splines (FKS). ReLU NNs and FKS span the same function space, and thus have the same theoretical expressivity. In the case of ReLU NNs, we show that their ill-conditioning degrades rapidly as the width of the network increases. This often leads to significantly poorer approximation in contrast to the FKS representation, which remains well-conditioned as the number of knots increases. We leverage the theory of optimal piecewise linear interpolants to improve the training procedure for a ReLU NN. Using the equidistribution principle, we propose a two-level procedure for training the FKS by first solving the nonlinear problem of finding the optimal knot locations of the interpolating FKS. Determining the optimal knots then acts as a good starting point for training the weights of the FKS. The training of the FKS gives insights into how we can train a ReLU NN effectively to give an equally accurate approximation. More precisely, we combine the training of the ReLU NN with an equidistribution based loss to find the breakpoints of the ReLU functions, combined with preconditioning the ReLU NN approximation (to take an FKS form) to find the scalings of the ReLU functions, leads to a well-conditioned and reliable method of finding an accurate ReLU NN approximation to a target function. We test this method on a series or regular, singular, and rapidly varying target functions and obtain good results realising the expressivity of the network in this case.

Best free knot linear spline approximation and its application to neural networks

Approximating NURBS Curves by Arc Splines

Equidistribution-based training of Free Knot Splines and ReLU Neural Networks

Mapping-to-Parameter Nonlinear Functional Regression with Novel B-spline Free Knot Placement Algorithm

Estimation of Knots in Linear Spline Models

Computing best discrete least-squares approximations by first-degree splines with free knots

Knot Calculation for Spline Fitting Via Sparse Optimization

Discrete approximation by first-degree splines with free knots

A discrete-time neural network for optimization problems with hybrid constraints.

Spline regression with automatic knot selection

On Knot Placement for Penalized Spline Regression

The Optimal Rate for Linear KB-splines and LKB-splines Approximation of High Dimensional Continuous Functions and its Application

SplineGen: Approximating Unorganized Points Through Generative AI

Effective Re-Parameterization and GA Based Knot Structure Optimization for High Quality T-spline Surface Fitting

Exact Penalty Method for Knot Selection of B-Spline Regression

SplineGen: a generative model for B-spline approximation of unorganized points

A comparison of deep networks with ReLU activation function and linear spline-type methods

B-Spline Recurrent Neurall Network and Its Application to Modelling

Approximate merging of B-spline curves via knot adjustment and constrained optimization

Learning and approximation capabilities of adaptive spline activation function neural networks

The limitation of neural nets for approximation and optimization