Abstract:Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies, (2) forecast long horizons, and (3) efficiently train over long sequences. State-space models (SSMs) are classical models for time series, and prior works combine SSMs with deep learning layers for efficient sequence modeling. However, we find fundamental limitations with these prior approaches, proving their SSM representations cannot express autoregressive time series processes. We thus introduce SpaceTime, a new state-space time series architecture that improves all three criteria. For expressivity, we propose a new SSM parameterization based on the companion matrix -- a canonical representation for discrete-time processes -- which enables SpaceTime's SSM layers to learn desirable autoregressive processes. For long horizon forecasting, we introduce a "closed-loop" variation of the companion SSM, which enables SpaceTime to predict many future time-steps by generating its own layer-wise inputs. For efficient training and inference, we introduce an algorithm that reduces the memory and compute of a forward pass with the companion matrix. With sequence length $\ell$ and state-space size $d$, we go from $\tilde{O}(d \ell)$ na\"ively to $\tilde{O}(d + \ell)$. In experiments, our contributions lead to state-of-the-art results on extensive and diverse benchmarks, with best or second-best AUROC on 6 / 7 ECG and speech time series classification, and best MSE on 14 / 16 Informer forecasting tasks. Furthermore, we find SpaceTime (1) fits AR($p$) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-the-art, and (3) speeds up training on real-world ETTh1 data by 73% and 80% relative wall-clock time over Transformers and LSTMs.

Towards a theory of learning dynamics in deep state space models

Deep Learning-based Approaches for State Space Models: A Selective Review

HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context

State Space Models on Temporal Graphs: A First-Principles Study

State Space Models as Foundation Models: A Control Theoretic Overview

Meta-Dynamical State Space Models for Integrative Neural Data Analysis

Theoretical Foundations of Deep Selective State-Space Models

Spectral State Space Models

Deep State Space Models for Nonlinear System Identification

Effectively Modeling Time Series with Simple Discrete State Spaces

Tuning Frequency Bias of State Space Models

State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?

State-space models can learn in-context by gradient descent

Physics-Informed Neural State Space Models via Learning and Evolution

Regularization-Based Efficient Continual Learning in Deep State-Space Models

Signal Extraction from Long-Term Ecological Data Using Bayesian and Non-Bayesian State-Space Models

Time Series Analysis by State Space Learning

Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models

Deep Direct Discriminative Decoders for High-dimensional Time-series Data Analysis

DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs

Learning Latent Dynamics via Invariant Decomposition and (Spatio-)Temporal Transformers