Abstract:Recent advances in deep learning have mainly relied on Transformers due to their data dependency and ability to learn at scale. The attention module in these architectures, however, exhibits quadratic time and space in input size, limiting their scalability for long-sequence modeling. Despite recent attempts to design efficient and effective architecture backbone for multi-dimensional data, such as images and multivariate time series, existing models are either data independent, or fail to allow inter- and intra-dimension communication. Recently, State Space Models (SSMs), and more specifically Selective State Space Models, with efficient hardware-aware implementation, have shown promising potential for long sequence modeling. Motivated by the success of SSMs, we present MambaMixer, a new architecture with data-dependent weights that uses a dual selection mechanism across tokens and channels, called Selective Token and Channel Mixer. MambaMixer connects selective mixers using a weighted averaging mechanism, allowing layers to have direct access to early features. As a proof of concept, we design Vision MambaMixer (ViM2) and Time Series MambaMixer (TSM2) architectures based on the MambaMixer block and explore their performance in various vision and time series forecasting tasks. Our results underline the importance of selective mixing across both tokens and channels. In ImageNet classification, object detection, and semantic segmentation tasks, ViM2 achieves competitive performance with well-established vision models and outperforms SSM-based vision models. In time series forecasting, TSM2 achieves outstanding performance compared to state-of-the-art methods while demonstrating significantly improved computational cost. These results show that while Transformers, cross-channel attention, and MLPs are sufficient for good performance in time series forecasting, neither is necessary.

MixMamba: Time Series Modeling with Adaptive Expertise

C-Mamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting.

CMMamba: Channel Mixing Mamba for Time Series Forecasting

CMamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting

Mamba or Transformer for Time Series Forecasting? Mixture of Universals (MoU) Is All You Need

ST-MoE: Spatio-Temporal Mixture of Experts for Multivariate Time Series Forecasting.

Bi-Mamba+: Bidirectional Mamba for Time Series Forecasting

UmambaTSF: A U-shaped Multi-Scale Long-Term Time Series Forecasting Method Using Mamba

Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting

Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics

TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting

Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting

A Mamba Foundation Model for Time Series Forecasting

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

SST: Multi-Scale Hybrid Mamba-Transformer Experts for Long-Short Range Time Series Forecasting

BlackMamba: Mixture of Experts for State-Space Models

MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection

FMamba: Mamba based on Fast-attention for Multivariate Time-series Forecasting

Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts

Is Mamba Effective for Time Series Forecasting?

Dynamic Mixture of Experts Models for Online Prediction