S2CrossMamba: Spatial–Spectral Cross-Mamba for Multimodal Remote Sensing Image Classification

Guanglian Zhang,Zhanxu Zhang,Jiangwei Deng,Lifeng Bian,Chen Yang

DOI: https://doi.org/10.1109/lgrs.2024.3488036

IF: 5.343

2024-01-01

IEEE Geoscience and Remote Sensing Letters

Abstract:Integration of complementary information from different modalities and efficient computation is crucial in remote sensing image classification applications. Convolutional neural networks (CNN), recurrent neural networks (RNN), and transformers have achieved great success in multimodal remote sensing classification, but their practical utility is constrained by the challenges of parallelization and the inefficiencies associated with overemphasis on computational complexity. To address these issues, we propose S 2 CrossMamba based on a state-space model (SSM), an efficient multimodal fusion network tailored for hyperspectral image (HSI) data. First, the Cross-SSM is designed to dynamically update the multimodal state information to fuse different modalities with all parameters learned multimodal input-dependent, yielding a centralized response. Secondly, the inverted bottleneck Cross-Mamba (IBCM) is designed based on Cross-SSM to increase the feature richness by mixing the multimodal features through the inverted bottleneck structure, and the mixed features are efficiently operated through Cross-SSM. To further enhance the HSI spatial and spectral feature extraction capability, dual-branch spectral Mamba and spatial Mamba are designed for feature fusion and learning based on spatial and spectral attention and IBCM structure. Our network achieves higher performance with a smaller number of parameters. The codes will be released at https://github.com/HyperSystemAndImageProc/S2CrossMamba.