Abstract:Remote sensing imagery has attracted significant attention in recent years due to its instrumental role in global environmental monitoring, land usage monitoring, and more. As image databases grow each year, performing automatic segmentation with deep learning models has gradually become the standard approach for processing the data. Despite the improved performance of current models, certain limitations remain unresolved. Firstly, training deep learning models for segmentation requires per-pixel annotations. Given the large size of datasets, only a small portion is fully annotated and ready for training. Additionally, the high intra-dataset variance in remote sensing data limits the transfer learning ability of such models. Although recently proposed generic segmentation models like SAM have shown promising results in zero-shot instance-level segmentation, adapting them to semantic segmentation is a non-trivial task. To tackle these challenges, we propose a novel method named Text2Seg for remote sensing semantic segmentation. Text2Seg overcomes the dependency on extensive annotations by employing an automatic prompt generation process using different visual foundation models (VFMs), which are trained to understand semantic information in various ways. This approach not only reduces the need for fully annotated datasets but also enhances the model's ability to generalize across diverse datasets. Evaluations on four widely adopted remote sensing datasets demonstrate that Text2Seg significantly improves zero-shot prediction performance compared to the vanilla SAM model, with relative improvements ranging from 31% to 225%. Our code is available at <a class="link-external link-https" href="https://github.com/Douglas2Code/Text2Seg" rel="external noopener nofollow">this https URL</a>.

Exploration of an Open Vocabulary Model on Semantic Segmentation for Street Scene Imagery

Auto-Vocabulary Semantic Segmentation

Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding

Open-RGBT: Open-vocabulary RGB-T Zero-shot Semantic Segmentation in Open-world Environments

Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Models

Exploring Simple Open-Vocabulary Semantic Segmentation

Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

GeoSAM: Fine-tuning SAM with Multi-Modal Prompts for Mobility Infrastructure Segmentation

SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints

Towards Open-Vocabulary Video Semantic Segmentation

Migratable urban street scene sensing method based on vision language pre-trained model

Boosting Segment Anything Model Towards Open-Vocabulary Learning

There is no SAMantics! Exploring SAM as a Backbone for Visual Understanding Tasks

LMSeg: Unleashing the Power of Large-Scale Models for Open-Vocabulary Semantic Segmentation

Multi-Modal Prototypes for Open-World Semantic Segmentation

RSAM-Seg: A SAM-based Approach with Prior Knowledge Integration for Remote Sensing Image Semantic Segmentation

Semantic and Instance Segmentation in Coastal Urban Spatial Perception: A Multi-Task Learning Framework with an Attention Mechanism

A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model

Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding

Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively