Recurrent neural networks: A constructive algorithm, and its properties

Ah Chung Tsoi,Shaohua Tan

DOI: https://doi.org/10.1016/S0925-2312(97)00011-8

IF: 6

1997-01-01

Neurocomputing

Abstract:In this paper, a constructive algorithm for a general recurrent neural network is proposed based on radial basis functions. It is shown that this algorithm is globally convergent. In addition, we will present two examples to illustrate the proposed method.

What problem does this paper attempt to address?

Analyze and Design Network Architectures by Recursion Formulas

Yilin Liao,Hao Wang,Zhaoran Liu,Haozhe Li,Xinggao Liu

2021-01-01

Abstract: The effectiveness of shortcut/skip-connection has been widely verified, which inspires massive explorations on neural architecture design. This work attempts to find an effective way to design new network architectures. It is discovered that the main difference between network architectures can be reflected in their recursion formulas. Based on this, a methodology is proposed to design novel network architectures from the perspective of mathematical formulas. Afterwards, a case study is provided to generate an improved architecture based on ResNet. Furthermore, the new architecture is compared with ResNet and then tested on ResNet-based networks. Massive experiments are conducted on CIFAR and ImageNet, which witnesses the significant performance improvements provided by the architecture.
A New Constructive Method to Optimize Neural Network Architecture and Generalization

Hou Muzhou,Moon Ho Lee

DOI: https://doi.org/10.48550/arXiv.1302.0324

2013-02-02

Neural and Evolutionary Computing

Abstract:In this paper, after analyzing the reasons of poor generalization and overfitting in neural networks, we consider some noise data as a singular value of a continuous function - jump discontinuity point. The continuous part can be approximated with the simplest neural networks, which have good generalization performance and optimal network architecture, by traditional algorithms such as constructive algorithm for feed-forward neural networks with incremental training, BP algorithm, ELM algorithm, various constructive algorithm, RBF approximation and SVM. At the same time, we will construct RBF neural networks to fit the singular value with every error in, and we prove that a function with jumping discontinuity points can be approximated by the simplest neural networks with a decay RBF neural networks in by each error, and a function with jumping discontinuity point can be constructively approximated by a decay RBF neural networks in by each error and the constructive part have no generalization influence to the whole machine learning system which will optimize neural network architecture and generalization performance, reduce the overfitting phenomenon by avoid fitting the noisy data.
Continuous Attractors of Discrete-Time Recurrent Neural Networks

Jiali Yu,Huajin Tang,Haizhou Li

DOI: https://doi.org/10.1007/s00521-012-0975-5

2012-01-01

Abstract:This paper studies the continuous attractors of discrete-time recurrent neural networks. Networks in discrete time can directly provide algorithms for efficient implementation in digital hardware. Continuous attractors of neural networks have been used to store and manipulate continuous stimuli for animals. A continuous attractor is defined as a connected set of stable equilibrium points. It forms a lower dimensional manifold in the original state space. Under some conditions, the complete analytical expressions for the continuous attractors of discrete-time linear recurrent neural networks as well as discrete-time linear-threshold recurrent neural networks are derived. Examples are employed to illustrate the theory.
Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem

Tang, H.J.,Tan, K.C.,Zhang, Y.

DOI: https://doi.org/10.1109/IJCNN.2002.1007530

2002-01-01

Abstract:We study the convergence of a class of discrete recurrent neural networks to solve linear variational inequality problem (LVIP). LVIP has important applications in engineering and economics. Not only the network's exponential convergence for the case of positive definite matrix is proved, but its global convergence for positive semidefinite matrix is also proved. Conditions are derived to guarantee the convergences of the network. Comprehensive examples are discussed and simulated to illustrate the results
The Learning Algorithm of Constructive NN

T Pin,Z Bo,Y Zhen

DOI: https://doi.org/10.1109/wcica.2002.1020859

2002-01-01

Abstract:We give a learning algorithm of a constructive neural network (CNN) which is a novel neural network model. With the advantages coming from the CNN, the learning algorithm can estimate and get the accurate pattern spatial distribution. First, an introduction to the CNN is given. Then the learning algorithm of the CNN is designed. Finally, the result of an example is shown and some suggestions are given.
Brain-Inspired Constructive Learning Algorithms with Evolutionally Additive Nonlinear Neurons

Le-Heng Fang,Wei Lin,Qiang Luo

DOI: https://doi.org/10.1142/s0218127418500682

2018-01-01

Abstract:In this article, inspired partially by the physiological evidence of brain’s growth and development, we developed a new type of constructive learning algorithm with evolutionally additive nonlinear neurons. The new algorithms have remarkable ability in effective regression and accurate classification. In particular, the algorithms are able to sustain a certain reduction of the loss function when the dynamics of the trained network are bogged down in the vicinity of the local minima. The algorithm augments the neural network by adding only a few connections as well as neurons whose activation functions are nonlinear, nonmonotonic, and self-adapted to the dynamics of the loss functions. Indeed, we analytically demonstrate the reduction dynamics of the algorithm for different problems, and further modify the algorithms so as to obtain an improved generalization capability for the augmented neural networks. Finally, through comparing with the classical algorithm and architecture for neural network construction, we show that our constructive learning algorithms as well as their modified versions have better performances, such as faster training speed and smaller network size, on several representative benchmark datasets including the MNIST dataset for handwriting digits.
A fast algorithm for solving large scale nonlinear optimization problems using RNN

Xiaoliang Xu,Huajin Tang,Xiaoxin Shi

DOI: https://doi.org/10.1109/ICCIS.2008.4670798

2008-01-01

Abstract:This paper presents a discrete-time recurrent neural network (RNN) model for solving nonlinear differentiable constrained optimization problems, which contain the special case of convex optimizations over constrained sets and variational inequality problem. The qualitative analysis results about the regularity and completeness of the proposed network have been obtained. It is shown that all trajectories starting from any initial point in Rfrn converge to the equilibrium set of the recurrent system. This RNN model shows its great simplicity in contrast to other existing neural network solvers. Simulations for a class of large scale linear complementarity problems illustrate the fast convergence and features of the proposed RNN model.
Discussion of Stability in a Class of Models on Recurrent Radial Basis Function Neural Networks

Meiqin Liu,Jida Chen

2000-01-01

Abstract:Based on radial basis function neural networks(RBFNNs) and recurrent neural networks (RNNs),a class of new models on recurrent radial basis function neural networks (RRBFNNs) are proposed.These new networks possess the advantages of RBFNNs and RNNs.In this paper,asymptotic stability and learning algorithms of RRBFNNs are researched,and some theorems and formulae are given.Simulation results show that RRBFNNs possess great potentiality in controlling unstable nonlinear system(e.g.the cart and inverted pendulum system).
A Recurrent Neural Network for Solving Nonconvex Optimization Problems

Xiaolin Hu,Jun Wang

DOI: https://doi.org/10.1109/ijcnn.2006.247077

2006-01-01

Abstract:An existing recurrent neural network for convex optimization is extended to solve nonconvex optimization problems. One of the prominent features of this neural network is the one-to-one correspondence between its equilibria and the Karush-Kuhn-Tucker (KKT) points of the nonconvex optimization problem. The conditions are derived under which the neural network (locally) converges to the KKT points. It is desired that the neural network is stable at minimum solutions, and unstable at maximum solutions or saddle solutions. It is found in the paper that most likely the neural network is unstable at the maximum solutions. Moreover, we found that if the derived conditions are not satisfied at minimum solutions, by transforming the original problem into an equivalent one with the p-power (or partial p-power) method, these conditions can be satisfied. As a result, the neural network will locally converge to a minimum solution. Finally, two illustrative examples are provided to demonstrate the performance of the recurrent neural network.
Interpreting recurrent neural networks behaviour via excitable network attractors

Andrea Ceni,Peter Ashwin,Lorenzo Livi

DOI: https://doi.org/10.1007/s12559-019-09634-2

2019-03-10

Abstract:Introduction: Machine learning provides fundamental tools both for scientific research and for the development of technologies with significant impact on society. It provides methods that facilitate the discovery of regularities in data and that give predictions without explicit knowledge of the rules governing a system. However, a price is paid for exploiting such flexibility: machine learning methods are typically black-boxes where it is difficult to fully understand what the machine is doing or how it is operating. This poses constraints on the applicability and explainability of such methods. Methods: Our research aims to open the black-box of recurrent neural networks, an important family of neural networks used for processing sequential data. We propose a novel methodology that provides a mechanistic interpretation of behaviour when solving a computational task. Our methodology uses mathematical constructs called excitable network attractors, which are invariant sets in phase space composed of stable attractors and excitable connections between them. Results and Discussion: As the behaviour of recurrent neural networks depends both on training and on inputs to the system, we introduce an algorithm to extract network attractors directly from the trajectory of a neural network while solving tasks. Simulations conducted on a controlled benchmark task confirm the relevance of these attractors for interpreting the behaviour of recurrent neural networks, at least for tasks that involve learning a finite number of stable states and transitions between them.

Machine Learning,Dynamical Systems
New Approach for Robust Stability Analysis of Delayed Recurrent Neural Networks

ZHANG Jian-hai,ZHANG Sen-lin,LIU Mei-qin

DOI: https://doi.org/10.3785/j.issn.1008-973x.2009.03.007

2009-01-01

Abstract:The standard neural network model(SNNM) was used to provide a unified framework for the stability analysis of various recurrent neural networks(RNN).By applying the Lyapunov stability theory and S-procedure technique,some sufficient stability conditions were derived for the global robust asymptotic and exponential stability of delayed SNNM with the form of linear matrix inequalities.The condition for robust exponential stability was formulated as a generalized eigenvalued problem,which established an estimation of the exponential convergence rate and improved the previous results.In the given examples,other RNNs were transformed into SNNM to be analyzed in a unified way.Simulation showed the effectiveness of the presented method and the validity of the sufficient conditions.
RCBNN:A New Constructing Method for Neural Network

XIE Zhen-Hua,LI Ning,SHANG Lin,CHEN Zhao-Qian,CHEN Shi-Fu

2005-01-01

Computer Science

Abstract:A good structure can greatly improve the power and convergent speed of neural network. According this, the constructing method is always a hotspot in neural network research. This paper proposes a constructing method for neural network based on Rough Set and Decision Tree. Rough Set has good capability of data analysis; Decision Tree is good at segmentation of continues-valued attributes. This method makes good use of these advantages. Experimental results show that the neural network designed by this method is easy constructible, good understandable, fast conver- gent, and small dimensional.
Recurrent Neural Network Approach Based On The Integral Representation Of The Drazin Inverse

Predrag S. Stanimirovic,Ivan S. Zivkovic,Yimin Wei

DOI: https://doi.org/10.1162/NECO_a_00771

IF: 3.278

2015-01-01

Neural Computation

Abstract:In this letter, we present the dynamical equation and corresponding artificial recurrent neural network for computing the Drazin inverse for arbitrary square real matrix, without any restriction on its eigenvalues. Conditions that ensure the stability of the defined recurrent neural network as well as its convergence toward the Drazin inverse are considered. Several illustrative examples present the results of computer simulations.
A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms

Danimir T. Doncevic,Alexander Mitsos,Yue Guo,Qianxiao Li,Felix Dietrich,Manuel Dahmen,Ioannis G. Kevrekidis

2023-07-06

Abstract:Meta-learning of numerical algorithms for a given task consists of the data-driven identification and adaptation of an algorithmic structure and the associated hyperparameters. To limit the complexity of the meta-learning problem, neural architectures with a certain inductive bias towards favorable algorithmic structures can, and should, be used. We generalize our previously introduced Runge-Kutta neural network to a recursively recurrent neural network (R2N2) superstructure for the design of customized iterative algorithms. In contrast to off-the-shelf deep learning approaches, it features a distinct division into modules for generation of information and for the subsequent assembly of this information towards a solution. Local information in the form of a subspace is generated by subordinate, inner, iterations of recurrent function evaluations starting at the current outer iterate. The update to the next outer iterate is computed as a linear combination of these evaluations, reducing the residual in this space, and constitutes the output of the network. We demonstrate that regular training of the weight parameters inside the proposed superstructure on input/output data of various computational problem classes yields iterations similar to Krylov solvers for linear equation systems, Newton-Krylov solvers for nonlinear equation systems, and Runge-Kutta integrators for ordinary differential equations. Due to its modularity, the superstructure can be readily extended with functionalities needed to represent more general classes of iterative algorithms traditionally based on Taylor series expansions.

Machine Learning,Numerical Analysis
An Augmented Lagrangian Method for Training Recurrent Neural Networks

Yue Wang,Chao Zhang,Xiaojun Chen

2024-08-20

Abstract:Recurrent Neural Networks (RNNs) are widely used to model sequential data in a wide range of areas, such as natural language processing, speech recognition, machine translation, and time series analysis. In this paper, we model the training process of RNNs with the ReLU activation function as a constrained optimization problem with a smooth nonconvex objective function and piecewise smooth nonconvex constraints. We prove that any feasible point of the optimization problem satisfies the no nonzero abnormal multiplier constraint qualification (NNAMCQ), and any local minimizer is a Karush-Kuhn-Tucker (KKT) point of the problem. Moreover, we propose an augmented Lagrangian method (ALM) and design an efficient block coordinate descent (BCD) method to solve the subproblems of the ALM. The update of each block of the BCD method has a closed-form solution. The stop criterion for the inner loop is easy to check and can be stopped in finite steps. Moreover, we show that the BCD method can generate a directional stationary point of the subproblem. Furthermore, we establish the global convergence of the ALM to a KKT point of the constrained optimization problem. Compared with the state-of-the-art algorithms, numerical results demonstrate the efficiency and effectiveness of the ALM for training RNNs.

Optimization and Control
A recurrent neural network for solving a class of general variational inequalities.

Xiaolin Hu,Jun Wang

DOI: https://doi.org/10.1109/TSMCB.2006.886166

2007-01-01

Abstract:This paper presents a recurrent neural-network model for solving a special class of general variational inequalities (GVIs), which includes classical VIs as special cases. It is proved that the proposed neural network (NN) for solving this class of GVIs can be globally convergent, globally asymptotically stable, and globally exponentially stable under different conditions. The proposed NN can be viewed as a modified version of the general projection NN existing in the literature. Several numerical examples are provided to demonstrate the effectiveness and performance of the proposed NN.
Explicit construction of recurrent neural networks effectively approximating discrete dynamical systems

Chikara Nakayama,Tsuyoshi Yoneda

2024-09-28

Abstract:We consider arbitrary bounded discrete time series originating from dynamical system with recursivity. More precisely, we provide an explicit construction of recurrent neural networks which effectively approximate the corresponding discrete dynamical systems.

Machine Learning,Dynamical Systems
Training Recurrent Neural Networks by Diffusion

Hossein Mobahi

DOI: https://doi.org/10.48550/arXiv.1601.04114

2016-02-05

Abstract:This work presents a new algorithm for training recurrent neural networks (although ideas are applicable to feedforward networks as well). The algorithm is derived from a theory in nonconvex optimization related to the diffusion equation. The contributions made in this work are two fold. First, we show how some seemingly disconnected mechanisms used in deep learning such as smart initialization, annealed learning rate, layerwise pretraining, and noise injection (as done in dropout and SGD) arise naturally and automatically from this framework, without manually crafting them into the algorithms. Second, we present some preliminary results on comparing the proposed method against SGD. It turns out that the new algorithm can achieve similar level of generalization accuracy of SGD in much fewer number of epochs.

Machine Learning
Recurrent Aggregators in Neural Algorithmic Reasoning

Kaijia Xu,Petar Veličković

2024-09-11

Abstract:Neural algorithmic reasoning (NAR) is an emerging field that seeks to design neural networks that mimic classical algorithmic computations. Today, graph neural networks (GNNs) are widely used in neural algorithmic reasoners due to their message passing framework and permutation equivariance. In this extended abstract, we challenge this design choice, and replace the equivariant aggregation function with a recurrent neural network. While seemingly counter-intuitive, this approach has appropriate grounding when nodes have a natural ordering -- and this is the case frequently in established reasoning benchmarks like CLRS-30. Indeed, our recurrent NAR (RNAR) model performs very strongly on such tasks, while handling many others gracefully. A notable achievement of RNAR is its decisive state-of-the-art result on the Heapsort and Quickselect tasks, both deemed as a significant challenge for contemporary neural algorithmic reasoners -- especially the latter, where RNAR achieves a mean micro-F1 score of 87%.

Machine Learning,Artificial Intelligence
Recurrent Neural Network for Computing the Drazin Inverse.

Predrag S Stanimirović,Ivan S Zivković,Yimin Wei

DOI: https://doi.org/10.1109/TNNLS.2015.2397551

IF: 14.255

2015-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This paper presents a recurrent neural network (RNN) for computing the Drazin inverse of a real matrix in real time. This recurrent neural network (RNN) is composed of n independent parts (subnetworks), where n is the order of the input matrix. These subnetworks can operate concurrently, so parallel and distributed processing can be achieved. In this way, the computational advantages over the existing sequential algorithms can be attained in real-time applications. The RNN defined in this paper is convenient for an implementation in an electronic circuit. The number of neurons in the neural network is the same as the number of elements in the output matrix, which represents the Drazin inverse. The difference between the proposed RNN and the existing ones for the Drazin inverse computation lies in their network architecture and dynamics. The conditions that ensure the stability of the defined RNN as well as its convergence toward the Drazin inverse are considered. In addition, illustrative examples and examples of application to the practical engineering problems are discussed to show the efficacy of the proposed neural network.

Recurrent neural networks: A constructive algorithm, and its properties

Analyze and Design Network Architectures by Recursion Formulas

A New Constructive Method to Optimize Neural Network Architecture and Generalization

Continuous Attractors of Discrete-Time Recurrent Neural Networks

Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem

The Learning Algorithm of Constructive NN

Brain-Inspired Constructive Learning Algorithms with Evolutionally Additive Nonlinear Neurons

A fast algorithm for solving large scale nonlinear optimization problems using RNN

Discussion of Stability in a Class of Models on Recurrent Radial Basis Function Neural Networks

A Recurrent Neural Network for Solving Nonconvex Optimization Problems

Interpreting recurrent neural networks behaviour via excitable network attractors

New Approach for Robust Stability Analysis of Delayed Recurrent Neural Networks

RCBNN:A New Constructing Method for Neural Network

Recurrent Neural Network Approach Based On The Integral Representation Of The Drazin Inverse

A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms

An Augmented Lagrangian Method for Training Recurrent Neural Networks

A recurrent neural network for solving a class of general variational inequalities.

Explicit construction of recurrent neural networks effectively approximating discrete dynamical systems

Training Recurrent Neural Networks by Diffusion

Recurrent Aggregators in Neural Algorithmic Reasoning

Recurrent Neural Network for Computing the Drazin Inverse.