Abstract:In this work, we investigate the structure and representation capacity of sinusoidal MLPs - multilayer perceptron networks that use sine as the activation function. These neural networks (known as neural fields) have become fundamental in representing common signals in computer graphics, such as images, signed distance functions, and radiance fields. This success can be primarily attributed to two key properties of sinusoidal MLPs: smoothness and compactness. These functions are smooth because they arise from the composition of affine maps with the sine function. This work provides theoretical results to justify the compactness property of sinusoidal MLPs and provides control mechanisms in the definition and training of these networks. We propose to study a sinusoidal MLP by expanding it as a harmonic sum. First, we observe that its first layer can be seen as a harmonic dictionary, which we call the input sinusoidal neurons. Then, a hidden layer combines this dictionary using an affine map and modulates the outputs using the sine, this results in a special dictionary of sinusoidal neurons. We prove that each of these sinusoidal neurons expands as a harmonic sum producing a large number of new frequencies expressed as integer linear combinations of the input frequencies. Thus, each hidden neuron produces the same frequencies, and the corresponding amplitudes are completely determined by the hidden affine map. We also provide an upper bound and a way of sorting these amplitudes that can control the resulting approximation, allowing us to truncate the corresponding series. Finally, we present applications for training and initialization of sinusoidal MLPs. Additionally, we show that if the input neurons are periodic, then the entire network will be periodic with the same period. We relate these periodic networks with the Fourier series representation.

Control Signal Generator Based on Ultra-High Frequency Polynomial and Trigonometric Higher Order Neural Networks

Data Classification Using Ultra-High Frequency SINC and Trigonometric Higher Order Neural Networks

Data Pattern Recognition Based on Ultra-High Frequency Sigmoid and Trigonometric Higher Order Neural Networks

Data Simulations Using Cosine and Sigmoid Higher Order Neural Networks

Models of Artificial Higher Order Neural Networks

Analogue Implementation of A Neural Network Controller for Ups Inverter Applications

Models of Artificial Multi-Polynomial Higher Order Neural Networks

Group Models of Artificial Polynomial and Trigonometric Higher Order Neural Networks

Neural-network-controlled single-phase UPS inverters with improved transient response and adaptability to various loads

Financial Data Prediction by Artificial Sine and Cosine Trigonometric Higher Order Neural Networks

Convergence Analysis and Digital Implementation of a Discrete-Time Neural Network for Model Predictive Control

SAS Nonlinear Models or Artificial Higher Order Neural Network Nonlinear Models?

Universal Neural Network Control of Mimo Uncertain Nonlinear Systems

Complete Stability Control on the High-Order Neural Networks

Adaptive neural network control for nonlinear uncertain systems with high-frequency disturbances

Control Signal Solving Model and Algorithm of Dynamic System Based on Process Neural Network

Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently

Fuzzy reinforced polynomial neural networks constructed with the aid of PNN architecture and fuzzy hybrid predictor based on nonlinear function

Nonlinear model predictive control using polynomial optimization methods

Artificial Higher Order Neural Networks for Economics and Business

Understanding Sinusoidal Neural Networks