Abstract:Researchers commonly believe that neural networks model a high-dimensional space but cannot give a clear definition of this space. What is this space? What is its dimension? And does it has finite dimensions? In this paper, we develop a plausible theory on interpreting neural networks in terms of the role of activation functions in neural networks and define a high-dimensional (more precisely, an infinite-dimensional) space that neural networks including deep-learning networks could create. We show that the activation function acts as a magnifying function that maps the low-dimensional linear space into an infinite-dimensional space, which can distinctly identify the polynomial approximation of any multivariate continuous function of the variable values being the same features of the given dataset. Given a dataset with each example of $d$ features $f_1$, $f_2$, $\cdots$, $f_d$, we believe that neural networks model a special space with infinite dimensions, each of which is a monomial $$\prod_{i_1, i_2, \cdots, i_d} f_1^{i_1} f_2^{i_2} \cdots f_d^{i_d}$$ for some non-negative integers ${i_1, i_2, \cdots, i_d} \in \mathbb{Z}_{0}^{+}=\{0,1,2,3,\ldots\} $. We term such an infinite-dimensional space a $\textit{ Super Space (SS)}$. We see such a dimension as the minimum information unit. Every neuron node previously through an activation layer in neural networks is a $\textit{ Super Plane (SP) }$, which is actually a polynomial of infinite degree. This $\textit{ Super Space }$ is something like a coordinate system, in which every multivalue function can be represented by a $\textit{ Super Plane }$. We also show that training NNs could at least be reduced to solving a system of nonlinear equations. %solve sets of nonlinear equations

Harmonic analysis of neural networks

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

Harmonic (Quantum) Neural Networks

Linear approximability of two-layer neural networks: A comprehensive analysis based on spectral decay

Fourier Neural Networks for Function Approximation

Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

Neural Networks, Hypersurfaces, and Radon Transforms

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Multistability and complete convergence analysis on high-order neural networks with a class of nonsmooth activation functions

Universal approximation with complex-valued deep narrow neural networks

Activation Functions Not To Active: A Plausible Theory on Interpreting Neural Networks

Approximation of RKHS Functionals by Neural Networks

Approximating nonlinear functions with latent boundaries in low-rank excitatory-inhibitory spiking networks

Optimal approximation using complex-valued neural networks

The Neural Network Model of Sinusoid Activation Transfer Function

Asymptotic Analysis of Neural Network Operators Employing the Hardy-Littlewood Maximal Inequality

Simultaneous approximation by neural network operators with applications to Voronovskaja formulas

Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems

Approximation Rates for Neural Networks with General Activation Functions