Spectral-Spatial Mamba for Hyperspectral Image Classification

Lingbo Huang,Yushi Chen,Xin He

2024-04-29

Abstract:Recently, deep learning models have achieved excellent performance in hyperspectral image (HSI) classification. Among the many deep models, Transformer has gradually attracted interest for its excellence in modeling the long-range dependencies of spatial-spectral features in HSI. However, Transformer has the problem of quadratic computational complexity due to the self-attention mechanism, which is heavier than other models and thus has limited adoption in HSI processing. Fortunately, the recently emerging state space model-based Mamba shows great computational efficiency while achieving the modeling power of Transformers. Therefore, in this paper, we make a preliminary attempt to apply the Mamba to HSI classification, leading to the proposed spectral-spatial Mamba (SS-Mamba). Specifically, the proposed SS-Mamba mainly consists of spectral-spatial token generation module and several stacked spectral-spatial Mamba blocks. Firstly, the token generation module converts any given HSI cube to spatial and spectral tokens as sequences. And then these tokens are sent to stacked spectral-spatial mamba blocks (SS-MB). Each SS-MB block consists of two basic mamba blocks and a spectral-spatial feature enhancement module. The spatial and spectral tokens are processed separately by the two basic mamba blocks, respectively. Besides, the feature enhancement module modulates spatial and spectral tokens using HSI sample's center region information. In this way, the spectral and spatial tokens cooperate with each other and achieve information fusion within each block. The experimental results conducted on widely used HSI datasets reveal that the proposed model achieves competitive results compared with the state-of-the-art methods. The Mamba-based method opens a new window for HSI classification.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper focuses on the classification problem of hyperspectral images (HSI). HSI has rich spatial and spectral information and is commonly used in environmental monitoring, precision agriculture, and other applications. In recent years, deep learning models have shown outstanding performance in HSI classification, especially the Transformer model, which is good at capturing long-range dependencies. However, the self-attention mechanism of Transformer leads to a quadratic increase in computational complexity, limiting its application in HSI processing.

The paper proposes a new method called Spectral-Spatial Mamba (SS-Mamba) to address the computational efficiency problem of Transformer. SS-Mamba adopts the Mamba structure based on the state space model, which maintains the long-range dependency modeling ability of Transformer while improving computational efficiency. The model consists of a spectral-spatial token generation module and stacked spectral-spatial Mamba blocks. The token generation module converts the HSI cube into spatial and spectral token sequences, which are then processed by the Mamba blocks. Each Mamba block contains two basic branches that respectively handle spatial and spectral features, and information fusion is achieved through a feature enhancement module.

Experimental results show that SS-Mamba is competitive with the current state-of-the-art methods on commonly used HSI datasets. This approach opens up new avenues for HSI classification, solves the problem of high computational complexity of Transformer, and makes deep learning analysis of hyperspectral images more efficient.