Abstract:From finance and healthcare to transportation and beyond, effective time series modeling underpins a wide range of applications. While transformers have achieved success, their reliance on global context limits scalability for lengthy sequences due to the quadratic increase in computational cost with sequence length. Recent research suggests linear models can achieve comparable performance with lower complexity. However, the heterogeneity and non-stationary characteristics of time series data continue to challenge single models’ ability to capture complex temporal dynamics, especially in long-term forecasting. This paper proposes MixMamba, a novel framework for time series modeling applicable across diverse domains. The framework leverages the content-based reasoning strengths of the Mamba model by integrating it as an expert within a mixture-of-experts (MoE) framework. This framework decomposes modeling into a pool of specialized experts, enabling the model to learn robust representations and capture the full spectrum of patterns present in time series data. Furthermore, a dynamic gating network is introduced within the framework. This network adaptively allocates each data segment to the most suitable expert based on its characteristics. This is crucial in non-stationary time series, as it allows the model to adjust dynamically to temporal changes in the underlying data distribution. To prevent bias towards a limited subset of experts, a load balancing loss function is incorporated. Extensive experiments on benchmark datasets demonstrate the effectiveness and robustness of our proposed method in various time series modeling tasks, including long-term and short-term forecasting, as well as classification.

Dynamic Mixture of Experts Models for Online Prediction

Bayesian estimation and prediction for certain mixtures

Mixture of Online and Offline Experts for Non-stationary Time Series

Prediction Sets for High-Dimensional Mixture of Experts Models

A similarity-based Bayesian mixture-of-experts model

Application of Mixture of Experts Model To Financial Time Series Forecasting

Adaptive sequential Monte Carlo by means of mixture of experts

A Mixture-of-Experts Prediction Framework for Evolutionary Dynamic Multiobjective Optimization

Semi-supervised Mixture of Latent Factor Analysis Models with Application to Online Key Variable Estimation

Dynamic Latent Class Model Averaging for Online Prediction

Fast deep mixtures of Gaussian process experts

Mixture ratio modeling of dynamic systems

MixMamba: Time Series Modeling with Adaptive Expertise

Functional Mixtures-of-Experts

A Deterministic Analysis of an Online Convex Mixture of Expert Algorithms

Continuous Prediction with Experts' Advice

Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process

Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models

Mixture Weight Estimation and Model Prediction in Multi-source Multi-target Domain Adaptation