Abstract:In this work, we investigate the structure and representation capacity of sinusoidal MLPs - multilayer perceptron networks that use sine as the activation function. These neural networks (known as neural fields) have become fundamental in representing common signals in computer graphics, such as images, signed distance functions, and radiance fields. This success can be primarily attributed to two key properties of sinusoidal MLPs: smoothness and compactness. These functions are smooth because they arise from the composition of affine maps with the sine function. This work provides theoretical results to justify the compactness property of sinusoidal MLPs and provides control mechanisms in the definition and training of these networks. We propose to study a sinusoidal MLP by expanding it as a harmonic sum. First, we observe that its first layer can be seen as a harmonic dictionary, which we call the input sinusoidal neurons. Then, a hidden layer combines this dictionary using an affine map and modulates the outputs using the sine, this results in a special dictionary of sinusoidal neurons. We prove that each of these sinusoidal neurons expands as a harmonic sum producing a large number of new frequencies expressed as integer linear combinations of the input frequencies. Thus, each hidden neuron produces the same frequencies, and the corresponding amplitudes are completely determined by the hidden affine map. We also provide an upper bound and a way of sorting these amplitudes that can control the resulting approximation, allowing us to truncate the corresponding series. Finally, we present applications for training and initialization of sinusoidal MLPs. Additionally, we show that if the input neurons are periodic, then the entire network will be periodic with the same period. We relate these periodic networks with the Fourier series representation.

The Neural Network Model of Sinusoid Activation Transfer Function

Understanding Sinusoidal Neural Networks

Fourier Neural Networks for Function Approximation

A New Neural Network Model And Its Application

Simo Fourier Neural Networks Research

Neural Network With Fourier Series-Based Transfer Functions for Filter Modeling

Fourier neural networks as function approximators and differential equation solvers

A Neural Network for Parameter Estimation of the Exponentially Damped Sinusoids.

Taming the Frequency Factory of Sinusoidal Networks

Sinusoidal Modulation Control Method in a Chaotic Neural Network

Complex-Valued Recurrent Neural Network with IIR Neuron Model: Training and Applications

Sine Neural Network (SNN) with Double-Stage Weights and Structure Determination (DS-WASD)

Multiple and Complete Stability of Recurrent Neural Networks With Sinusoidal Activation Function

A Weights-Direct-Determination Method and Structure-Automatic-Determination Algorithm for SIMO Trigonometrically-Activated Fourier Neural Networks

Training via quantum superposition circumventing local minima and vanishing gradient of sinusoidal neural network

Application of Feedforward Neural Network with Different Activation Characteristic's Neurons in SINS Alignment

A New Artificial Neuron Proposal with Trainable Simultaneous Local and Global Activation Function

An FHN-HR Neuron Network Coupled With a Novel Locally Active Memristor and Its DSP Implementation

Complex-exponential Fourier Neuronal Network and Its Hidden-Neuron Growing Algorithm

Hopf bifurcation and chaos in a single delayed neuron equation with non-monotonic activation function

A Novel Fuzzy Neural Network Approach for Excitation System Modeling