Abstract:Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers in large language modeling, offering linear scaling with sequence length and improved training efficiency. However, LRNNs struggle to perform state-tracking which may impair performance in tasks such as code evaluation or tracking a chess game. Even parity, the simplest state-tracking task, which non-linear RNNs like LSTM handle effectively, cannot be solved by current LRNNs. Recently, Sarrof et al. (2024) demonstrated that the failure of LRNNs like Mamba to solve parity stems from restricting the value range of their diagonal state-transition matrices to $[0, 1]$ and that incorporating negative values can resolve this issue. We extend this result to non-diagonal LRNNs, which have recently shown promise in models such as DeltaNet. We prove that finite precision LRNNs with state-transition matrices having only positive eigenvalues cannot solve parity, while complex eigenvalues are needed to count modulo $3$. Notably, we also prove that LRNNs can learn any regular language when their state-transition matrices are products of identity minus vector outer product matrices, each with eigenvalues in the range $[-1, 1]$. Our empirical results confirm that extending the eigenvalue range of models like Mamba and DeltaNet to include negative values not only enables them to solve parity but consistently improves their performance on state-tracking tasks. Furthermore, pre-training LRNNs with an extended eigenvalue range for language modeling achieves comparable performance and stability while showing promise on code and math data. Our work enhances the expressivity of modern LRNNs, broadening their applicability without changing the cost of training or inference.

Regularizing RNNs by Stabilizing Activations

RNN algorithm optimization based on extended unsaturated region

Recurrent Neural Network Regularization

Variance regularization of RNNLM for speech recognition

Enhance the Performance of Deep Neural Networks Via L2 Regularization on the Input of Activations.

An Investigation on Deep Learning with Beta Stabilizer

Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues

Regularized Flexible Activation Function Combinations for Deep Neural Networks

Occam's Gates

Improving RNN transducer with normalized jointer network

CTC Regularized Model Adaptation for Improving LSTM RNN Based Multi-Accent Mandarin Speech Recognition

Stabilising and accelerating light gated recurrent units for automatic speech recognition

Alignment Restricted Streaming Recurrent Neural Network Transducer.

Nonlinear Regularization Decoding Method for Speech Recognition

State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions

Stabilizing RNN Gradients through Pre-training

Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training

Explicit Regularization via Regularizer Mirror Descent

Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace

Regularizing Deep Networks Using Efficient Layerwise Adversarial Training

Regularized Structured Perceptron: A Case Study on Chinese Word Segmentation, POS Tagging and Parsing.