Abstract:Semantic segmentation of remote sensing imagery plays a pivotal role in extracting precise information for diverse down-stream applications. Recent development of the Segment Anything Model (SAM), an advanced general-purpose segmentation model, has revolutionized this field, presenting new avenues for accurate and efficient segmentation. However, SAM is limited to generating segmentation results without class information. Consequently, the utilization of such a powerful general vision model for semantic segmentation in remote sensing images has become a focal point of research. In this paper, we present a streamlined framework aimed at leveraging the raw output of SAM by exploiting two novel concepts called SAM-Generated Object (SGO) and SAM-Generated Boundary (SGB). More specifically, we propose a novel object loss and further introduce a boundary loss as augmentative components to aid in model optimization in a general semantic segmentation framework. Taking into account the content characteristics of SGO, we introduce the concept of object consistency to leverage segmented regions lacking semantic information. By imposing constraints on the consistency of predicted values within objects, the object loss aims to enhance semantic segmentation performance. Furthermore, the boundary loss capitalizes on the distinctive features of SGB by directing the model's attention to the boundary information of the object. Experimental results on two well-known datasets, namely ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness of our proposed method. The source code for this work will be accessible at <a class="link-external link-https" href="https://github.com/sstary/SSRS" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve The paper aims to address two main issues in remote sensing image semantic segmentation: 1. **Lack of Category Information**: The existing Segment Anything Model (SAM) can generate high-quality segmentation masks, but these masks lack category information. This means that although SAM can identify objects well, it cannot provide specific category labels for each object. 2. **Inaccurate Boundaries and Over-Segmentation**: Current semantic segmentation methods often suffer from inaccurate boundaries and over-segmentation when predicting segmentation maps. This affects the quality and usability of the segmentation results. To tackle these issues, the authors propose a new framework that utilizes the raw outputs generated by SAM (referred to as SAM-Generated Object (SGO) and SAM-Generated Boundary (SGB)) and introduces two new loss functions—Object Consistency Loss and Boundary Preservation Loss—to improve segmentation performance. ### Key Points of the Solution 1. **Object Consistency Loss**: Enhances the performance of semantic segmentation by ensuring consistent pixel values within the same object. Specifically, this loss function is implemented by calculating the mean squared error between the predicted values and the average value within the object region. 2. **Boundary Preservation Loss**: Uses the detailed boundary information in SGB to guide the model to better focus on the boundaries of objects. This loss function evaluates the accuracy of boundary detection by calculating the boundary F1 score between the segmentation output and the ground truth boundaries. 3. **Simple and Effective Framework**: This framework does not require complex design or adjustments to general semantic segmentation models, nor does it need to generate pseudo labels, thus simplifying the integration and optimization process of the model. ### Experimental Results Experiments were conducted on two well-known datasets, ISPRS Vaihingen and LoveDA Urban, and the results showed significant improvements in the performance of multiple semantic segmentation models. In particular, the effective combination of Object Consistency Loss and Boundary Preservation Loss significantly improved the accuracy and boundary quality of the segmentation results. ### Main Contributions 1. **Proposed a New Framework for Efficient Utilization of SGO and SGB**, highlighting the value and effectiveness of SAM's raw outputs. 2. **Introduced Object Consistency Loss and Boundary Preservation Loss**, marking the first time SAM's raw outputs are directly utilized in semantic segmentation tasks without the need for additional category hints. 3. **Extensive Experimental Validation** demonstrates that this method can be widely applied to different datasets and general models, showing high practicality and scalability. Through these innovations, this research provides new solutions for the field of remote sensing image semantic segmentation and is expected to promote further development in this area.

SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints

SAM-Assisted Remote Sensing Imagery Semantic Segmentation With Object and Boundary Constraints

MeSAM: Multiscale Enhanced Segment Anything Model for Optical Remote Sensing Images

Remote sensing image semantic segmentation via class-guided structural interaction and boundary perception

RSAM-Seg: A SAM-based Approach with Prior Knowledge Integration for Remote Sensing Image Semantic Segmentation

RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM

Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Models

SAM-RSIS: Progressively Adapting SAM With Box Prompting to Remote Sensing Image Instance Segmentation

A Creative Weak Supervised Semantic Segmentation for Remote Sensing Images

Evaluating the Efficacy of Segment Anything Model for Delineating Agriculture and Urban Green Spaces in Multiresolution Aerial and Spaceborne Remote Sensing Images

Self-guided Few-shot Semantic Segmentation for Remote Sensing Imagery Based on Large Vision Models

The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot

SCD-SAM: Adapting Segment Anything Model for Semantic Change Detection in Remote Sensing Imagery

SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Model

A deep learning based framework for remote sensing image ground object segmentation

PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Images

Incorporating DeepLabv3+and Object-Based Image Analysis for Semantic Segmentation of Very High Resolution Remote Sensing Images

GeoSAM: Fine-tuning SAM with Multi-Modal Prompts for Mobility Infrastructure Segmentation

Semantic Attention and Structured Model for Weakly Supervised Instance Segmentation in Optical and SAR Remote Sensing Imagery

Integrated Framework for Unsupervised Building Segmentation with Segment Anything Model-Based Pseudo-Labeling and Weakly Supervised Learning

A boundary enhancement loss function for semantic segmentation of land cover