Abstract:Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies, (2) forecast long horizons, and (3) efficiently train over long sequences. State-space models (SSMs) are classical models for time series, and prior works combine SSMs with deep learning layers for efficient sequence modeling. However, we find fundamental limitations with these prior approaches, proving their SSM representations cannot express autoregressive time series processes. We thus introduce SpaceTime, a new state-space time series architecture that improves all three criteria. For expressivity, we propose a new SSM parameterization based on the companion matrix -- a canonical representation for discrete-time processes -- which enables SpaceTime's SSM layers to learn desirable autoregressive processes. For long horizon forecasting, we introduce a "closed-loop" variation of the companion SSM, which enables SpaceTime to predict many future time-steps by generating its own layer-wise inputs. For efficient training and inference, we introduce an algorithm that reduces the memory and compute of a forward pass with the companion matrix. With sequence length $\ell$ and state-space size $d$, we go from $\tilde{O}(d \ell)$ na\"ively to $\tilde{O}(d + \ell)$. In experiments, our contributions lead to state-of-the-art results on extensive and diverse benchmarks, with best or second-best AUROC on 6 / 7 ECG and speech time series classification, and best MSE on 14 / 16 Informer forecasting tasks. Furthermore, we find SpaceTime (1) fits AR($p$) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-the-art, and (3) speeds up training on real-world ETTh1 data by 73% and 80% relative wall-clock time over Transformers and LSTMs.

Coupling LSTM neural networks and state-space models through analytically tractable inference

Relational State-Space Model for Stochastic Multi-Object Systems

A Bayesian Long Short-Term Memory Model for Value at Risk and Expected Shortfall Joint Forecasting

Effective LSTMs with Seasonal-Trend Decomposition and Adaptive Learning and Niching-Based Backtracking Search Algorithm for Time Series Forecasting

Stanza: A Nonlinear State Space Model for Probabilistic Inference in Non-Stationary Time Series

Forward-Backward Latent State Inference for Hidden Continuous-Time semi-Markov Chains

Effectively Modeling Time Series with Simple Discrete State Spaces

Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting

A Hybrid Spiking Neurons Embedded LSTM Network for Multivariate Time Series Learning under Concept-Drift Environment

Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting

Convolutional State Space Models for Long-Range Spatiotemporal Modeling

Stochastic Gradient MCMC for Nonlinear State Space Models

Improving the Performance of the LSTM and HMM Model via Hybridization

Learning Sparse Hidden States In Long Short-Term Memory

Incorporating Exponential Smoothing into MLP: A Simple but Effective Sequence Model

Compositional simulation-based inference for time series

Shared state space model for background information extraction and time series prediction

Synergetic Learning of Heterogeneous Temporal Sequences for Multi-Horizon Probabilistic Forecasting

Maximum approximate likelihood estimation of general continuous-time state-space models

ABC-based Forecasting in State Space Models

A Point Mass Proposal Method for Bayesian State-Space Model Fitting