MetaSegNet: Metadata-collaborative Vision-Language Representation Learning for Semantic Segmentation of Remote Sensing Images

Libo Wang,Sijun Dong,Ying Chen,Xiaoliang Meng,Shenghui Fang,Songlin Fei

DOI: https://doi.org/10.1109/TGRS.2024.3477548

2024-10-30

Abstract:Semantic segmentation of remote sensing images plays a vital role in a wide range of Earth Observation applications, such as land use land cover mapping, environment monitoring, and sustainable development. Driven by rapid developments in artificial intelligence, deep learning (DL) has emerged as the mainstream for semantic segmentation and has achieved many breakthroughs in the field of remote sensing. However, most DL-based methods focus on unimodal visual data while ignoring rich multimodal information involved in the real world. Non-visual data, such as text, can gather extra knowledge from the real world, which can strengthen the interpretability, reliability, and generalization of visual models. Inspired by this, we propose a novel metadata-collaborative segmentation network (MetaSegNet) that applies vision-language representation learning for semantic segmentation of remote sensing images. Unlike the common model structure that only uses unimodal visual data, we extract the key characteristic (e.g. the climate zone) from freely available remote sensing image metadata and transfer it into geographic text prompts via the generic ChatGPT. Then, we construct an image encoder, a text encoder, and a crossmodal attention fusion subnetwork to extract the image and text feature and apply image-text interaction. Benefiting from such a design, the proposed MetaSegNet not only demonstrates superior generalization in zero-shot testing but also achieves competitive accuracy with the state-of-the-art semantic segmentation methods on the large-scale OpenEarthMap dataset (70.4% mIoU) and the Potsdam dataset (93.3% mean F1 score) as well as the LoveDA dataset (52.0% mIoU).

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is:

In the task of semantic segmentation of remote sensing images, most existing deep learning-based methods mainly focus on unimodal visual data, neglecting the rich multimodal information in the real world. Non-visual data (such as text) can collect additional knowledge from the real world, which can enhance the interpretability, reliability, and generalization ability of visual models. To this end, the paper proposes a new metadata collaborative segmentation network (MetaSegNet) to achieve semantic segmentation of remote sensing images through visual-language representation learning.

Specifically, the main contributions of the paper include:
1. Developing a new metadata collaborative inference framework that utilizes a multimodal visual-language architecture. To the best of the authors' knowledge, this is the first time that freely available image metadata has been mined and transformed into knowledge-based geographic text prompts to improve the reliability of remote sensing segmentation models.
2. Designing a ChatGPT-based pipeline that can generate professional geographic text prompts by querying specific climatic zone geographic object features extracted from image metadata.
3. Designing a plug-and-play cross-modal attention fusion decoder (CAFDecoder) that can effectively integrate heterogeneous image-text features, enhancing intra-modal and inter-modal dependencies, and enriching high-level semantic content.

Through these innovations, MetaSegNet not only performs well in zero-shot testing but also achieves comparable accuracy to existing state-of-the-art semantic segmentation methods on large-scale datasets such as OpenEarthMap (70.4% mIoU), Potsdam (93.3% average F1 score), and LoveDA (52.0% mIoU).