Abstract:Complex-valued neural networks (CVNNs) have recently shown promising empirical success, for instance for increasing the stability of recurrent neural networks and for improving the performance in tasks with complex-valued inputs, such as in MRI fingerprinting. While the overwhelming success of Deep Learning in the real-valued case is supported by a growing mathematical foundation, such a foundation is still largely lacking in the complex-valued case. We thus analyze the expressivity of CVNNs by studying their approximation properties. Our results yield the first quantitative approximation bounds for CVNNs that apply to a wide class of activation functions including the popular modReLU and complex cardioid activation functions. Precisely, our results apply to any activation function that is smooth but not polyharmonic on some non-empty open set; this is the natural generalization of the class of smooth and non-polynomial activation functions to the complex setting. Our main result shows that the error for the approximation of $C^k$-functions scales as $m^{-k/(2n)}$ for $m \to \infty$ where $m$ is the number of neurons, $k$ the smoothness of the target function and $n$ is the (complex) input dimension. Under a natural continuity assumption, we show that this rate is optimal; we further discuss the optimality when dropping this assumption. Moreover, we prove that the problem of approximating $C^k$-functions using continuous approximation methods unavoidably suffers from the curse of dimensionality.

Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Why Deep Neural Networks for Function Approximation?

Approximating smooth functions by deep neural networks with sigmoid activation function

Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations

Deep vs. shallow networks : An approximation theory perspective

Approximation smooth and sparse functions by deep neural networks without saturation

SMU: smooth activation function for deep networks using smoothing maximum technique

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

Approximation and interpolation of deep neural networks

Uniform Approximation with Quadratic Neural Networks

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Neural networks with ReLU powers need less depth

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function

Sampling Complexity of Deep Approximation Spaces

Deep neural network approximation of composite functions without the curse of dimensionality

On the Expressive Power of Neural Networks

On the approximation of rough functions with deep neural networks

Optimal approximation using complex-valued neural networks