Explicit construction of recurrent neural networks effectively approximating discrete dynamical systems

Chikara Nakayama,Tsuyoshi Yoneda

2024-09-28

Abstract:We consider arbitrary bounded discrete time series originating from dynamical system with recursivity. More precisely, we provide an explicit construction of recurrent neural networks which effectively approximate the corresponding discrete dynamical systems.

Machine Learning,Dynamical Systems

What problem does this paper attempt to address?

Exponential Attractor of Kappa-Almost Periodic Sequence Solution of Discrete-Time Bidirectional Neural Networks

Zhenkun Huang

DOI: https://doi.org/10.1016/j.simpat.2009.11.007

IF: 4.199

2009-01-01

Simulation Modelling Practice and Theory

Abstract:In this paper, several criteria are obtained for the existence and exponential attractivity of a unique κ-almost periodic sequence solution of discrete-time bidirectional neural networks (BNNs). Our results generalize the corresponding results about almost periodic sequence solution in common sense. Finally, computer simulations illustrate the dynamic behavior of discrete-time bidirectional neural networks. It is shown that discretization step κ affects dynamical characteristics of bidirectional neural networks and our results on κ-almost periodic sequence solution lead to effective estimates of continuous-time neural networks during real-time computations.
Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Guoliang Dong,Jingyi Wang,Jun Sun,Yang Zhang,Xinyu Wang,Ting Dai,Jin Song Dong,Xingen Wang

DOI: https://doi.org/10.1145/3324884.3416592

2020-01-01

Abstract:Neural networks are becoming a popular tool for solving many real-world problems such as object recognition and machine translation, thanks to its exceptional performance as an end-to-end solution. However, neural networks are complex black-box models, which hinders humans from interpreting and consequently trusting them in making critical decisions. Towards interpreting neural networks, several approaches have been proposed to extract simple deterministic models from neural networks. The results are not encouraging (e.g., low accuracy and limited scalability), fundamentally due to the limited expressiveness of such simple models. In this work, we propose an approach to extract probabilistic automata for interpreting an important class of neural networks, i.e., recurrent neural networks. Our work distinguishes itself from existing approaches in two important ways. One is that probability is used to compensate for the loss of expressiveness. This is inspired by the observation that human reasoning is often `probabilistic'. The other is that we adaptively identify the right level of abstraction so that a simple model is extracted in a request-specific way. We conduct experiments on several real-world datasets using state-of-the-art architectures including GRU and LSTM. The result shows that our approach significantly improves existing approaches in terms of accuracy or scalability. Lastly, we demonstrate the usefulness of the extracted models through detecting adversarial texts.
Recurrent Neural Networks for Recursive Identification of Nonlinear Dynamic Process

Jia-Wen Dong,Ji-xin Qian,You-Xian Sun

DOI: https://doi.org/10.1109/icit.1994.467030

1994-01-01

Abstract:In this paper, modified Elman-type recurrent neural networks (1990) were developed to identify the dynamic nonlinear systems with generalised backpropagation recursive algorithm. Analysis shows that introduction of adjustable self-connections of context units provides network ability to model high order input-output mapping, unbiased estimates can be achieved without the need to fit additive noise model. An industrial application example shows its efficiency
Residual Recurrent Neural Networks for Learning Sequential Representations.

Boxuan Yue,Junwei Fu,Jun Liang

DOI: https://doi.org/10.3390/info9030056

2018-01-01

Information

Abstract:Recurrent neural networks (RNN) are efficient in modeling sequences for generation and classification, but their training is obstructed by the vanishing and exploding gradient issues. In this paper, we reformulate the RNN unit to learn the residual functions with reference to the hidden state instead of conventional gated mechanisms such as long short-term memory (LSTM) and the gated recurrent unit (GRU). The residual structure has two main highlights: firstly, it solves the gradient vanishing and exploding issues for large time-distributed scales; secondly, the residual structure promotes the optimizations for backward updates. In the experiments, we apply language modeling, emotion classification and polyphonic modeling to evaluate our layer compared with LSTM and GRU layers. The results show that our layer gives state-of-the-art performance, outperforms LSTM and GRU layers in terms of speed, and supports an accuracy competitive with that of the other methods.
A Constructive Approach to Function Realization by Neural Stochastic Differential Equations

Tanya Veeravalli,Maxim Raginsky

2023-09-22

Abstract:The problem of function approximation by neural dynamical systems has typically been approached in a top-down manner: Any continuous function can be approximated to an arbitrary accuracy by a sufficiently complex model with a given architecture. This can lead to high-complexity controls which are impractical in applications. In this paper, we take the opposite, constructive approach: We impose various structural restrictions on system dynamics and consequently characterize the class of functions that can be realized by such a system. The systems are implemented as a cascade interconnection of a neural stochastic differential equation (Neural SDE), a deterministic dynamical system, and a readout map. Both probabilistic and geometric (Lie-theoretic) methods are used to characterize the classes of functions realized by such systems.

Optimization and Control,Machine Learning
Continuous Attractors of Discrete-Time Recurrent Neural Networks

Jiali Yu,Huajin Tang,Haizhou Li

DOI: https://doi.org/10.1007/s00521-012-0975-5

2012-01-01

Abstract:This paper studies the continuous attractors of discrete-time recurrent neural networks. Networks in discrete time can directly provide algorithms for efficient implementation in digital hardware. Continuous attractors of neural networks have been used to store and manipulate continuous stimuli for animals. A continuous attractor is defined as a connected set of stable equilibrium points. It forms a lower dimensional manifold in the original state space. Under some conditions, the complete analytical expressions for the continuous attractors of discrete-time linear recurrent neural networks as well as discrete-time linear-threshold recurrent neural networks are derived. Examples are employed to illustrate the theory.
High-order neural network structures for identification of dynamical systems

E.B. Kosmatopoulos,M.M. Polycarpou,M.A. Christodoulou,P.A. Ioannou

DOI: https://doi.org/10.1109/72.363477

1995-03-01

IEEE Transactions on Neural Networks

Abstract:Several continuous-time and discrete-time recurrent neural network models have been developed and applied to various engineering problems. One of the difficulties encountered in the application of recurrent networks is the derivation of efficient learning algorithms that also guarantee the stability of the overall system. This paper studies the approximation and learning properties of one class of recurrent networks, known as high-order neural networks; and applies these architectures to the identification of dynamical systems. In recurrent high-order neural networks, the dynamic components are distributed throughout the network in the form of dynamic neurons. It is shown that if enough high-order connections are allowed then this network is capable of approximating arbitrary dynamical systems. Identification schemes based on high-order network architectures are designed and analyzed.

English Else
Stimulus-Driven and Spontaneous Dynamics in Excitatory-Inhibitory Recurrent Neural Networks for Sequence Representation

Alfred Rajakumar,John Rinzel,Zhe S Chen

DOI: https://doi.org/10.1162/neco_a_01418

2021-09-16

Abstract:Recurrent neural networks (RNNs) have been widely used to model sequential neural dynamics ("neural sequences") of cortical circuits in cognitive and motor tasks. Efforts to incorporate biological constraints and Dale's principle will help elucidate the neural representations and mechanisms of underlying circuits. We trained an excitatory-inhibitory RNN to learn neural sequences in a supervised manner and studied the representations and dynamic attractors of the trained network. The trained RNN was robust to trigger the sequence in response to various input signals and interpolated a time-warped input for sequence representation. Interestingly, a learned sequence can repeat periodically when the RNN evolved beyond the duration of a single sequence. The eigenspectrum of the learned recurrent connectivity matrix with growing or damping modes, together with the RNN's nonlinearity, were adequate to generate a limit cycle attractor. We further examined the stability of dynamic attractors while training the RNN to learn two sequences. Together, our results provide a general framework for understanding neural sequence representation in the excitatory-inhibitory RNN.
Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies

Dominik Schmidt,Georgia Koppe,Zahra Monfared,Max Beutelspacher,Daniel Durstewitz

DOI: https://doi.org/10.48550/arXiv.1910.03471

2021-03-12

Abstract:A main theoretical interest in biology and physics is to identify the nonlinear dynamical system (DS) that generated observed time series. Recurrent Neural Networks (RNNs) are, in principle, powerful enough to approximate any underlying DS, but in their vanilla form suffer from the exploding vs. vanishing gradients problem. Previous attempts to alleviate this problem resulted either in more complicated, mathematically less tractable RNN architectures, or strongly limited the dynamical expressiveness of the RNN. Here we address this issue by suggesting a simple regularization scheme for vanilla RNNs with ReLU activation which enables them to solve long-range dependency problems and express slow time scales, while retaining a simple mathematical structure which makes their DS properties partly analytically accessible. We prove two theorems that establish a tight connection between the regularized RNN dynamics and its gradients, illustrate on DS benchmarks that our regularization approach strongly eases the reconstruction of DS which harbor widely differing time scales, and show that our method is also en par with other long-range architectures like LSTMs on several tasks.

Machine Learning,Quantitative Methods
Adaptive deep density approximation for stochastic dynamical systems

Junjie He,Qifeng Liao,Xiaoliang Wan

2024-05-05

Abstract:In this paper we consider adaptive deep neural network approximation for stochastic dynamical systems. Based on the Liouville equation associated with the stochastic dynamical systems, a new temporal KRnet (tKRnet) is proposed to approximate the probability density functions (PDFs) of the state variables. The tKRnet gives an explicit density model for the solution of the Liouville equation, which alleviates the curse of dimensionality issue that limits the application of traditional grid based numerical methods. To efficiently train the tKRnet, an adaptive procedure is developed to generate collocation points for the corresponding residual loss function, where samples are generated iteratively using the approximate density function at each iteration. A temporal decomposition technique is also employed to improve the long-time integration. Theoretical analysis of our proposed method is provided, and numerical examples are presented to demonstrate its performance.

Numerical Analysis,Machine Learning
Recurrent Neural Network Approach Based On The Integral Representation Of The Drazin Inverse

Predrag S. Stanimirovic,Ivan S. Zivkovic,Yimin Wei

DOI: https://doi.org/10.1162/NECO_a_00771

IF: 3.278

2015-01-01

Neural Computation

Abstract:In this letter, we present the dynamical equation and corresponding artificial recurrent neural network for computing the Drazin inverse for arbitrary square real matrix, without any restriction on its eigenvalues. Conditions that ensure the stability of the defined recurrent neural network as well as its convergence toward the Drazin inverse are considered. Several illustrative examples present the results of computer simulations.
Feed-forward approximations to dynamic recurrent network architectures

Dylan Richard Muir

DOI: https://doi.org/10.1162/neco_a_01042

2017-09-15

Abstract:Recurrent neural network architectures can have useful computational properties, with complex temporal dynamics and input-sensitive attractor states. However, evaluation of recurrent dynamic architectures requires solution of systems of differential equations, and the number of evaluations required to determine their response to a given input can vary with the input, or can be indeterminate altogether in the case of oscillations or instability. In feed-forward networks, by contrast, only a single pass through the network is needed to determine the response to a given input. Modern machine-learning systems are designed to operate efficiently on feed-forward architectures. We hypothesised that two-layer feedforward architectures with simple, deterministic dynamics could approximate the responses of single-layer recurrent network architectures. By identifying the fixed-point responses of a given recurrent network, we trained two-layer networks to directly approximate the fixed-point response to a given input. These feed-forward networks then embodied useful computations, including competitive interactions, information transformations and noise rejection. Our approach was able to find useful approximations to recurrent networks, which can then be evaluated in linear and deterministic time complexity.

Neural and Evolutionary Computing,Neurons and Cognition
Impulsive Control for Existence, Uniqueness, and Global Stability of Periodic Solutions of Recurrent Neural Networks with Discrete and Continuously Distributed Delays.

Xiaodi Li,Shiji Song

DOI: https://doi.org/10.1109/tnnls.2012.2236352

IF: 14.255

2013-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:In this paper, a class of recurrent neural networks with discrete and continuously distributed delays is considered. Sufficient conditions for the existence, uniqueness, and global exponential stability of a periodic solution are obtained by using contraction mapping theorem and stability theory on impulsive functional differential equations. The proposed method, which differs from the existing results in the literature, shows that network models may admit a periodic solution which is globally exponentially stable via proper impulsive control strategies even if it is originally unstable or divergent. Two numerical examples and their computer simulations are offered to show the effectiveness of our new results.
Systematic construction of continuous-time neural networks for linear dynamical systems

Chinmay Datar,Adwait Datar,Felix Dietrich,Wil Schilders

2024-03-25

Abstract:Discovering a suitable neural network architecture for modeling complex dynamical systems poses a formidable challenge, often involving extensive trial and error and navigation through a high-dimensional hyper-parameter space. In this paper, we discuss a systematic approach to constructing neural architectures for modeling a subclass of dynamical systems, namely, Linear Time-Invariant (LTI) systems. We use a variant of continuous-time neural networks in which the output of each neuron evolves continuously as a solution of a first-order or second-order Ordinary Differential Equation (ODE). Instead of deriving the network architecture and parameters from data, we propose a gradient-free algorithm to compute sparse architecture and network parameters directly from the given LTI system, leveraging its properties. We bring forth a novel neural architecture paradigm featuring horizontal hidden layers and provide insights into why employing conventional neural architectures with vertical hidden layers may not be favorable. We also provide an upper bound on the numerical errors of our neural networks. Finally, we demonstrate the high accuracy of our constructed networks on three numerical examples.

Machine Learning,Dynamical Systems,Numerical Analysis
Kernel Limit of Recurrent Neural Networks Trained on Ergodic Data Sequences

Samuel Chun-Hei Lam,Justin Sirignano,Konstantinos Spiliopoulos

2024-05-15

Abstract:Mathematical methods are developed to characterize the asymptotics of recurrent neural networks (RNN) as the number of hidden units, data samples in the sequence, hidden state updates, and training steps simultaneously grow to infinity. In the case of an RNN with a simplified weight matrix, we prove the convergence of the RNN to the solution of an infinite-dimensional ODE coupled with the fixed point of a random algebraic equation. The analysis requires addressing several challenges which are unique to RNNs. In typical mean-field applications (e.g., feedforward neural networks), discrete updates are of magnitude $\mathcal{O}(\frac{1}{N})$ and the number of updates is $\mathcal{O}(N)$. Therefore, the system can be represented as an Euler approximation of an appropriate ODE/PDE, which it will converge to as $N \rightarrow \infty$. However, the RNN hidden layer updates are $\mathcal{O}(1)$. Therefore, RNNs cannot be represented as a discretization of an ODE/PDE and standard mean-field techniques cannot be applied. Instead, we develop a fixed point analysis for the evolution of the RNN memory states, with convergence estimates in terms of the number of update steps and the number of hidden units. The RNN hidden layer is studied as a function in a Sobolev space, whose evolution is governed by the data sequence (a Markov chain), the parameter updates, and its dependence on the RNN hidden layer at the previous time step. Due to the strong correlation between updates, a Poisson equation must be used to bound the fluctuations of the RNN around its limit equation. These mathematical methods give rise to the neural tangent kernel (NTK) limits for RNNs trained on data sequences as the number of data samples and size of the neural network grow to infinity.

Machine Learning,Probability
Recurrent neural networks for dynamical systems: Applications to ordinary differential equations, collective motion, and hydrological modeling

K. Gajamannage,D. I. Jayathilake,Y. Park,E. M. Bollt

DOI: https://doi.org/10.1063/5.0088748

2023-01-06

Abstract:Classical methods of solving spatiotemporal dynamical systems include statistical approaches such as autoregressive integrated moving average, which assume linear and stationary relationships between systems' previous outputs. Development and implementation of linear methods are relatively simple, but they often do not capture non-linear relationships in the data. Thus, artificial neural networks (ANNs) are receiving attention from researchers in analyzing and forecasting dynamical systems. Recurrent neural networks (RNNs), derived from feed-forward ANNs, use internal memory to process variable-length sequences of inputs. This allows RNNs to be applicable for finding solutions for a vast variety of problems in spatiotemporal dynamical systems. Thus, in this paper, we utilize RNNs to treat some specific issues associated with dynamical systems. Specifically, we analyze the performance of RNNs applied to three tasks: reconstruction of correct Lorenz solutions for a system with a formulation error, reconstruction of corrupted collective motion trajectories, and forecasting of streamflow time series possessing spikes, representing three fields, namely, ordinary differential equations, collective motion, and hydrological modeling, respectively. We train and test RNNs uniquely in each task to demonstrate the broad applicability of RNNs in the reconstruction and forecasting the dynamics of dynamical systems.

mathematics, applied,physics, mathematical
Inverse Approximation Theory for Nonlinear Recurrent Neural Networks

Shida Wang,Zhong Li,Qianxiao Li

2024-02-06

Abstract:We prove an inverse approximation theorem for the approximation of nonlinear sequence-to-sequence relationships using recurrent neural networks (RNNs). This is a so-called Bernstein-type result in approximation theory, which deduces properties of a target function under the assumption that it can be effectively approximated by a hypothesis space. In particular, we show that nonlinear sequence relationships that can be stably approximated by nonlinear RNNs must have an exponential decaying memory structure - a notion that can be made precise. This extends the previously identified curse of memory in linear RNNs into the general nonlinear setting, and quantifies the essential limitations of the RNN architecture for learning sequential relationships with long-term memory. Based on the analysis, we propose a principled reparameterization method to overcome the limitations. Our theoretical results are confirmed by numerical experiments. The code has been released in <a class="link-external link-https" href="https://github.com/radarFudan/Curse-of-memory" rel="external noopener nofollow">this https URL</a>

Machine Learning,Artificial Intelligence,Dynamical Systems
Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems

Yifei Duan,Li'ang Li,Guanghua Ji,Yongqiang Cai

2024-10-27

Abstract:Deep learning has made significant applications in the field of data science and natural science. Some studies have linked deep neural networks to dynamic systems, but the network structure is restricted to the residual network. It is known that residual networks can be regarded as a numerical discretization of dynamic systems. In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems, where the width of the network is equal to the dimension of the input and output. Our proof is based on the properties of the leaky-ReLU function and the numerical technique of splitting method to solve differential equations. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks.

Machine Learning,Numerical Analysis
Implicit recurrent networks: A novel approach to stationary input processing with recurrent neural networks in deep learning

Sebastian Sanokowski

DOI: https://doi.org/10.48550/arXiv.2010.10564

2020-10-21

Abstract:The brain cortex, which processes visual, auditory and sensory data in the brain, is known to have many recurrent connections within its layers and from higher to lower layers. But, in the case of machine learning with neural networks, it is generally assumed that strict feed-forward architectures are suitable for static input data, such as images, whereas recurrent networks are required mainly for the processing of sequential input, such as language. However, it is not clear whether also processing of static input data benefits from recurrent connectivity. In this work, we introduce and test a novel implementation of recurrent neural networks with lateral and feed-back connections into deep learning. This departure from the strict feed-forward structure prevents the use of the standard error backpropagation algorithm for training the networks. Therefore we provide an algorithm which implements the backpropagation algorithm on a implicit implementation of recurrent networks, which is different from state-of-the-art implementations of recurrent neural networks. Our method, in contrast to current recurrent neural networks, eliminates the use of long chains of derivatives due to many iterative update steps, which makes learning computationally less costly. It turns out that the presence of recurrent intra-layer connections within a one-layer implicit recurrent network enhances the performance of neural networks considerably: A single-layer implicit recurrent network is able to solve the XOR problem, while a feed-forward network with monotonically increasing activation function fails at this task. Finally, we demonstrate that a two-layer implicit recurrent architecture leads to a better performance in a regression task of physical parameters from the measured trajectory of a damped pendulum.

Machine Learning,Neural and Evolutionary Computing

Explicit construction of recurrent neural networks effectively approximating discrete dynamical systems

Exponential Attractor of Kappa-Almost Periodic Sequence Solution of Discrete-Time Bidirectional Neural Networks

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Recurrent Neural Networks for Recursive Identification of Nonlinear Dynamic Process

Residual Recurrent Neural Networks for Learning Sequential Representations.

A Constructive Approach to Function Realization by Neural Stochastic Differential Equations

Continuous Attractors of Discrete-Time Recurrent Neural Networks

High-order neural network structures for identification of dynamical systems

Stimulus-Driven and Spontaneous Dynamics in Excitatory-Inhibitory Recurrent Neural Networks for Sequence Representation

Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies

Adaptive deep density approximation for stochastic dynamical systems

Recurrent Neural Network Approach Based On The Integral Representation Of The Drazin Inverse

Feed-forward approximations to dynamic recurrent network architectures

Impulsive Control for Existence, Uniqueness, and Global Stability of Periodic Solutions of Recurrent Neural Networks with Discrete and Continuously Distributed Delays.

Systematic construction of continuous-time neural networks for linear dynamical systems

Kernel Limit of Recurrent Neural Networks Trained on Ergodic Data Sequences

Recurrent neural networks for dynamical systems: Applications to ordinary differential equations, collective motion, and hydrological modeling

Inverse Approximation Theory for Nonlinear Recurrent Neural Networks

Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems

Implicit recurrent networks: A novel approach to stationary input processing with recurrent neural networks in deep learning