AMMUNet: Multi-Scale Attention Map Merging for Remote Sensing Image Segmentation

Yang Yang,Shunyi Zheng

2024-04-20

Abstract:The advancement of deep learning has driven notable progress in remote sensing semantic segmentation. Attention mechanisms, while enabling global modeling and utilizing contextual information, face challenges of high computational costs and require window-based operations that weaken capturing long-range dependencies, hindering their effectiveness for remote sensing image processing. In this letter, we propose AMMUNet, a UNet-based framework that employs multi-scale attention map merging, comprising two key innovations: the granular multi-head self-attention (GMSA) module and the attention map merging mechanism (AMMM). GMSA efficiently acquires global information while substantially mitigating computational costs in contrast to global multi-head self-attention mechanism. This is accomplished through the strategic utilization of dimension correspondence to align granularity and the reduction of relative position bias parameters, thereby optimizing computational efficiency. The proposed AMMM effectively combines multi-scale attention maps into a unified representation using a fixed mask template, enabling the modeling of global attention mechanism. Experimental evaluations highlight the superior performance of our approach, achieving remarkable mean intersection over union (mIoU) scores of 75.48\% on the challenging Vaihingen dataset and an exceptional 77.90\% on the Potsdam dataset, demonstrating the superiority of our method in precise remote sensing semantic segmentation. Codes are available at

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily addresses the challenges in the task of semantic segmentation of remote sensing images, particularly the high computational cost associated with attention mechanisms and the weakened ability to capture long-distance dependencies due to window operations. To tackle these issues, the authors propose the AMMUNet framework.

AMMUNet is an architecture based on U-Net, which employs ResNet in the encoder part to extract multi-scale features and introduces two key innovative modules in the decoder part:

1. **Granular Multi-Head Self-Attention (GMSA) Module**: This module divides the input feature map into smaller 2×2 pixel sub-regions and computes queries, keys, and values within each sub-region. This approach effectively captures global information while significantly reducing the computational cost compared to the global multi-head self-attention mechanism.

2. **Attention Map Merging Mechanism (AMMM)**: This mechanism uses a fixed mask template to efficiently merge attention maps at different scales into a unified representation, enabling the model to capture global attention mechanisms.

Through these innovations, AMMUNet can efficiently capture both local and global contextual information in remote sensing images, thereby improving segmentation accuracy. Experimental results show that AMMUNet achieves significant mean Intersection over Union (mIoU) scores on the Vaihingen and Potsdam datasets, with 75.48% and 77.90% respectively, demonstrating its superiority in precise remote sensing semantic segmentation.