Abstract:Recurrent neural networks (RNNs) are powerful models for processing time-series data, but it remains challenging to understand how they function. Improving this understanding is of substantial interest to both the machine learning and neuroscience communities. The framework of reverse engineering a trained RNN by linearizing around its fixed points has provided insight, but the approach has significant challenges. These include difficulty choosing which fixed point to expand around when studying RNN dynamics and error accumulation when reconstructing the nonlinear dynamics with the linearized dynamics. We present a new model that overcomes these limitations by co-training an RNN with a novel switching linear dynamical system (SLDS) formulation. A first-order Taylor series expansion of the co-trained RNN and an auxiliary function trained to pick out the RNN's fixed points govern the SLDS dynamics. The results are a trained SLDS variant that closely approximates the RNN, an auxiliary function that can produce a fixed point for each point in state-space, and a trained nonlinear RNN whose dynamics have been regularized such that its first-order terms perform the computation, if possible. This model removes the post-training fixed point optimization and allows us to unambiguously study the learned dynamics of the SLDS at any point in state-space. It also generalizes SLDS models to continuous manifolds of switching points while sharing parameters across switches. We validate the utility of the model on two synthetic tasks relevant to previous work reverse engineering RNNs. We then show that our model can be used as a drop-in in more complex architectures, such as LFADS, and apply this LFADS hybrid to analyze single-trial spiking activity from the motor system of a non-human primate.

The Power of Linear Recurrent Neural Networks

Residual Recurrent Neural Networks for Learning Sequential Representations.

Learning The Sequential Temporal Information with Recurrent Neural Networks

Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies

Nonlinear recurrent neural networks for finite-time solution of general time-varying linear matrix equations.

Linear RNNs Provably Learn Linear Dynamical Systems

Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems

Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction

A locally linear RBF network-based state-dependent AR model for nonlinear time series modeling

Real-time Forecasting of Time Series in Financial Markets Using Sequentially Trained Many-to-one LSTMs

AR-Net: A simple Auto-Regressive Neural Network for time-series

Robust Recurrent Neural Networks for Time Series Forecasting.

Exploring the Promise and Limits of Real-Time Recurrent Learning

Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues

Recurrent neural networks for dynamical systems: Applications to ordinary differential equations, collective motion, and hydrological modeling

Approximation to Nonlinear Discrete-Time Systems by Recurrent Neural Networks.

Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory

Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions

Learning Long-Term Dependencies in Irregularly-Sampled Time Series

Zhang Neural Network for Linear Time-Varying Equation Solving and Its Robotic Application