Abstract:Artificial neural networks are a computational framework that has become a focus of widespread interest. One of the most widely used neural networks is the feedforward neural network (FNN). This type of neural network can be used to learn the underlying rules from examples. This learning ability enables FNNs to have wide applicability. However, the theory behind this neural network model is still immature. There are many deficiencies of the current neural network learning algorithms that have hindered their usefulness. In this dissertation, we surveyed the research in FNN learning. Several new algorithms are proposed to improve the learning efficiency of FNNs. We have developed a globally guided neural network training algorithm that converges to a global optimal solution and reduces the training time. Both stochastic and deterministic global optimization approaches are employed for neural network training. The stochastic methods include genetic algorithms, simulated annealing, and pure random searches. Deterministic methods considered for neural net training are branch-and-bound based Lipschitz optimizations. By exploring the special structure of the FNN and the property of the sigmoid activation function, we developed procedures for computing Lipschitz constants over subsets of the weight space. With local Lipschitz constants we can identify weight regions that do not contain promising solutions and develop pruning methods that reduce the search space. The main advantage of the global optimal training algorithms (GOTA) is that they yield a guaranteed global optimal solution. GOTA can also be combined with local search procedures, such as backpropagation, to produce more efficient, but still globally convergent algorithms.

Dynamical Optimal Learning For Fnn And Its Applications

New Dynamical Optimal Learning for Linear Multilayer FNN

Dynamic node creation and fast learning algorithm for a hybrid feedforward neural network

A New Dynamic Optimal Learning Rate for a Two-Layer Neural Network

Optimal Training of Feedforward Neural Networks Using Teaching-Learning-Based Optimization with Modified Learning Phases

Learning complex nonlinear dynamics of a 3D translational parallel manipulator using neural network

An Advanced NN Learning Algorithm Based on DFP Method

Optimal Algorithm for a Multilayer Feedforward Net and Its Application

Convergence Rates of Training Deep Neural Networks Via Alternating Minimization Methods.

A mixture varying-gain dynamic learning network for solving nonlinear and nonconvex constrained optimization problems

Globally Convergent Neural Networks

A Novel Learning Algorithm For Dynamic Self-Organizing Fuzzy Neural Network

A Novel Neural Network Learning Method for Dynamically Tuning Regularization Coefficient

Dynamic Learning Rate for Neural Networks: A Fixed-Time Stability Approach

An Efficient Self-Organizing Deep Fuzzy Neural Network for Nonlinear System Modeling

A Diffusion-Neural-Network for Learning from Small Samples

Towards Hyperparameter-Agnostic DNN Training via Dynamical System Insights

TDN: Twice-Least-Square Double-Parallel Neural Networks.

Network Iterative Learning for Dynamic Deep Neural Networks via Morphism

A Fast Learning Algorithm for Feedfoward Neural Network Based on the Layer-by-Layer and Neuron-by-Neuron Optimizing Procedure