Abstract:Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies, (2) forecast long horizons, and (3) efficiently train over long sequences. State-space models (SSMs) are classical models for time series, and prior works combine SSMs with deep learning layers for efficient sequence modeling. However, we find fundamental limitations with these prior approaches, proving their SSM representations cannot express autoregressive time series processes. We thus introduce SpaceTime, a new state-space time series architecture that improves all three criteria. For expressivity, we propose a new SSM parameterization based on the companion matrix -- a canonical representation for discrete-time processes -- which enables SpaceTime's SSM layers to learn desirable autoregressive processes. For long horizon forecasting, we introduce a "closed-loop" variation of the companion SSM, which enables SpaceTime to predict many future time-steps by generating its own layer-wise inputs. For efficient training and inference, we introduce an algorithm that reduces the memory and compute of a forward pass with the companion matrix. With sequence length $\ell$ and state-space size $d$, we go from $\tilde{O}(d \ell)$ na\"ively to $\tilde{O}(d + \ell)$. In experiments, our contributions lead to state-of-the-art results on extensive and diverse benchmarks, with best or second-best AUROC on 6 / 7 ECG and speech time series classification, and best MSE on 14 / 16 Informer forecasting tasks. Furthermore, we find SpaceTime (1) fits AR($p$) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-the-art, and (3) speeds up training on real-world ETTh1 data by 73% and 80% relative wall-clock time over Transformers and LSTMs.

StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization

SMR: State Memory Replay for Long Sequence Modeling

HOPE for a Robust Parameterization of Long-memory State Space Models

Mathematical Formalism for Memory Compression in Selective State Space Models

Provable Benefits of Complex Parameterizations for Structured State Space Models

LongSSM: On the Length Extension of State-space Models in Language Modelling

Longhorn: State Space Models are Amortized Online Learners

Integrating symmetry of environment by designing special basis functions for value function approximation in reinforcement learning

Tuning Frequency Bias of State Space Models

Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models

Parameter-Efficient Fine-Tuning of State Space Models

The Illusion of State in State-Space Models

Schrodinger's Memory: Large Language Models

Incorporating Exponential Smoothing into MLP: A Simple but Effective Sequence Model

Efficiently Modeling Long Sequences with Structured State Spaces

From Generalization Analysis to Optimization Designs for State Space Models

Effectively Modeling Time Series with Simple Discrete State Spaces

Liquid Structural State-Space Models

Intermediate-term memory mechanism inspired lightweight single image super resolution

Theoretical Foundations of Deep Selective State-Space Models

Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning