Abstract:The radial basis function (RBF) networks are one of the most widely used models for function approximation in the regression problem. In the learning paradigm, the best approximation is recursively or iteratively searched for based on observed data (teacher signals). One encounters difficulties in such a process when two component basis functions become identical, or when the magnitude of one component becomes null. In this case, the number of the components reduces by one, and then the reduced component recovers as the learning process proceeds further, provided such a component is necessary for the best approximation. Strange behaviors, especially the plateau phenomena, have been observed in dynamics of learning when such reduction occurs. There exist singularities in the space of parameters, and the above reduction takes place at the singular regions. This paper focuses on a detailed analysis of the dynamical behaviors of learning near the overlap and elimination singularities in RBF networks, based on the averaged learning equation that is applicable to both on-line and batch mode learning. We analyze the stability on the overlap singularity by solving the eigenvalues of the Hessian explicitly. Based on the stability analysis, we plot the analytical dynamic vector fields near the singularity, which are then compared to those real trajectories obtained by a numeric method. We also confirm the existence of the plateaus in both batch and on-line learning by simulation.

Theoretical and Numerical Analysis of Learning Dynamics Near Singularity in Multilayer Perceptrons

Influence Area of Overlap Singularity in Multilayer Perceptrons

Application of the Error Function in Analyzing the Learning Dynamics Near Singularities of the Multilayer Perceptrons

Dynamics of learning near singularities in layered networks.

Multiperiodicity of Periodically Oscillated Discrete-Time Neural Networks with Transient Excitatory Self-Connections and Sigmoidal Nonlinearities

Online Learning Dynamics of Radial Basis Function Neural Networks Near the Singularity

Dynamics of learning near singularities in radial basis function networks.

Averaged Learning Equations of Error-Function-Based Multilayer Perceptrons

Stability analysis of opposite singularity in multilayer perceptrons

Dynamics of Learning In Hierarchical Models – Singularity and Milnor Attractor

Numerical Analysis Near Singularities in RBF Networks.

Analytical Form Of Fisher Information Matrix Of Bipoloar-Activation-Function-Based Multilayer Perceptrons

Eigenvalue Analysis on Singularity in RBF Networks

Stability Analysis of Recurrent Multilayer Perceptrons:Lmi Approach

Stability and Bifurcation Analysis in a Class of Two-Neuron Networks with Resonant Bilinear Terms

Learning Time-Scales in Two-Layers Neural Networks

Singularities in the Parameter Identification of Nonlinear Dynamic Systems

On the learning dynamics of two-layer quadratic neural networks for understanding deep learning

Consideration on the learning efficiency of multiple-layered neural networks with linear units

Fisher Information Matrix of Unipolar Activation Function-Based Multilayer Perceptrons

Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron