Multi-Granularity Aggregation Network for Remote Sensing Few-Shot Segmentation

Shi-Feng Peng,Guo-Sen Xie,Fang Zhao,Xiangbo Shu,Qingshan Liu
DOI: https://doi.org/10.1109/tgrs.2024.3503273
IF: 8.2
2024-01-01
IEEE Transactions on Geoscience and Remote Sensing
Abstract:Few-shot semantic segmentation (FSS) aims to segment a query image using a limited number of densely annotated support images from the same category. Most existing conventional FSS methods are tailored for coping with images from natural scenarios. Unlike natural images, remote sensing images usually have a similar background context among the support and query image pairs, and more severe intra-class inconsistency exists due to overhead shooting views. However, facing such realistic and challenging remote sensing FSS tasks, the existing methods seldom consider these intrinsic characteristics from a unified viewpoint, thus leading to inferior results. To solve the above dilemma, we propose a multi-granularity aggregation network (MGANet) to progressively capture multi-granularity discriminative information, for tackling the remote sensing FSS task. Specifically, MGANet consists of a multi-granularity similarity (MGS) module and an adaptive multi-prototype aggregation (AMPA) module. To fully utilize background context, MGS extracts multi-granularity support and query feature maps from the backbone network to calculate a holistic correlation by incorporating the background information. Next, to alleviate the intra-class inconsistency of remote sensing images, AMPA decomposes the support foreground region into mainstay and auxiliary sub-regions by the guidance of reverse prediction on support features, thus generating three types of prototypes by masked average pooling on these paired features and masks. Furthermore, these multi-prototypes are collaboratively interacted with the query features to pursue reinforced discriminative features, relying on prototype-aware slot attention. Extensive experiments on iSAID-5 i and LoveDA-2 i demonstrate well the superiority of the proposed MGANet. The source code is available at https://github.com/CVL-hub/MGANet/.
What problem does this paper attempt to address?