Pan-Mamba: Effective pan-sharpening with State Space Model

Xuanhua He,Ke Cao,Keyu Yan,Rui Li,Chengjun Xie,Jie Zhang,Man Zhou

2024-02-19

Abstract:Pan-sharpening involves integrating information from lowresolution multi-spectral and high-resolution panchromatic images to generate high-resolution multi-spectral counterparts. While recent advancements in the state space model, particularly the efficient long-range dependency modeling achieved by Mamba, have revolutionized computer vision community, its untapped potential in pan-sharpening motivates our exploration. Our contribution, Pan-Mamba, represents a novel pansharpening network that leverages the efficiency of the Mamba model in global information modeling. In Pan-Mamba, we customize two core components: channel swapping Mamba and cross-modal Mamba, strategically designed for efficient cross-modal information exchange and fusion. The former initiates a lightweight cross-modal interaction through the exchange of partial panchromatic and multispectral channels, while the latter facilities the information representation capability by exploiting inherent cross-modal relationships. Through extensive experiments across diverse datasets, our proposed approach surpasses state-of-theart methods, showcasing superior fusion results in pan-sharpening. To the best of our knowledge, this work is the first attempt in exploring the potential of the Mamba model and establishes a new frontier in the pan-sharpening techniques. The source code is available at https://github.com/alexhe101/Pan-Mamba .

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve

The paper aims to address the issue of pansharpening in the fusion of multispectral images (MS) and panchromatic images (PAN). Specifically, the paper proposes a novel pansharpening network named **Pan-Mamba**, leveraging the advantages of the Mamba model in global information modeling. Pan-Mamba includes the following core components:

1. **Mamba Block**: Used to extract long-range dependencies in PAN and LRMS images.
2. **Channel Exchange Mamba Block**: Achieves lightweight cross-modal interaction by exchanging parts of the PAN and MS channels.
3. **Cross-Modal Mamba Block**: Utilizes inherent cross-modal relationships for information fusion, filtering redundant features.

These components collectively achieve efficient feature extraction and fusion, thereby surpassing existing state-of-the-art methods on multiple public remote sensing datasets, demonstrating stronger spectral accuracy and texture information retention capabilities.