Abstract:The essence of multi-modal fusion lies in exploiting the complementary information inherent in diverse modalities. However, prevalent fusion methods rely on traditional neural architectures and are inadequately equipped to capture the dynamics of interactions across modalities, particularly in presence of complex intra- and inter-modality correlations. Recent advancements in State Space Models (SSMs), notably exemplified by the Mamba model, have emerged as promising contenders. Particularly, its state evolving process implies stronger modality fusion paradigm, making multi-modal fusion on SSMs an appealing direction. However, fusing multiple modalities is challenging for SSMs due to its hardware-aware parallelism designs. To this end, this paper proposes the Coupled SSM model, for coupling state chains of multiple modalities while maintaining independence of intra-modality state processes. Specifically, in our coupled scheme, we devise an inter-modal hidden states transition scheme, in which the current state is dependent on the states of its own chain and that of the neighbouring chains at the previous time-step. To fully comply with the hardware-aware parallelism, we devise an expedite coupled state transition scheme and derive its corresponding global convolution kernel for parallelism. Extensive experiments on CMU-MOSEI, CH-SIMS, CH-SIMSV2 through multi-domain input verify the effectiveness of our model compared to current state-of-the-art methods, improved F1-Score by 0.4\%, 0.9\%, and 2.3\% on the three datasets respectively, 49\% faster inference and 83.7\% GPU memory save. The results demonstrate that Coupled Mamba model is capable of enhanced multi-modal fusion.

Multiple-Model Estimation With Heterogeneous State Representation

State Estimation with Missing Measurements Using IMM

Multiple adaptive factors based interacting multiple model estimator

Interacting multiple model algorithm based on H∞ filter

On Multiple-Model Extended Target Multi-Bernoulli Filters

An efficient algorithm unifying MHT and Multiple Model approaches

Interacting Multiple Model-based Joint Homography Matrix and Multiple Object State Estimation

IMM Fusion Estimation with Multiple Asynchronous Sensors.

Second-Order Markov Chain Based Multiple-Model Algorithm for Maneuvering Target Tracking

Gaussian Mixture Multiple-Model Multi-Bernoulli Filters For Nonlinear Models Via Unscented Transforms

Multi-Scan Multi-Sensor Multi-Object State Estimation

Accurate and Robust State Estimation via Fusion of Visual-Inertial-UWB with Time Synchronization

Advanced State Estimation for Multi-Articulated Virtual Track Trains: A Fusion Approach

Estimation of multiple networks in Gaussian mixture models.

Design of the adaptive interacting multiple model algorithm

The Multiple Model Poisson Multi-Bernoulli Mixture Filter for Extended Target Tracking

Filtering of nonlinear systems with measurement loss by RUKF-IMM

IMM Algorithm Based on H∞ Filter for Maneuvering Target Tracking

Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model

Multiple Models Information Fusion Algorithm for Cooperative Localization

State estimation for networked systems: an extended IMM algorithm