Abstract:Although most existing multi-modal salient object detection (SOD) methods demonstrate effectiveness through training models from scratch, the limited multi-modal data hinders these methods from reaching optimality. In this paper, we propose a novel framework to explore and exploit the powerful feature representation and zero-shot generalization ability of the pre-trained Segment Anything Model (SAM) for multi-modal SOD. Despite serving as a recent vision fundamental model, driving the class-agnostic SAM to comprehend and detect salient objects accurately is non-trivial, especially in challenging scenes. To this end, we develop \underline{SAM} with se\underline{m}antic f\underline{e}ature fu\underline{s}ion guidanc\underline{e} (Sammese), which incorporates multi-modal saliency-specific knowledge into SAM to adapt SAM to multi-modal SOD tasks. However, it is difficult for SAM trained on single-modal data to directly mine the complementary benefits of multi-modal inputs and comprehensively utilize them to achieve accurate saliency prediction. To address these issues, we first design a multi-modal complementary fusion module to extract robust multi-modal semantic features by integrating information from visible and thermal or depth image pairs. Then, we feed the extracted multi-modal semantic features into both the SAM image encoder and mask decoder for fine-tuning and prompting, respectively. Specifically, in the image encoder, a multi-modal adapter is proposed to adapt the single-modal SAM to multi-modal information. In the mask decoder, a semantic-geometric prompt generation strategy is proposed to produce corresponding embeddings with various saliency cues. Extensive experiments on both RGB-D and RGB-T SOD benchmarks show the effectiveness of the proposed framework. The code will be available at \url{<a class="link-external link-https" href="https://github.com/Angknpng/Sammese" rel="external noopener nofollow">this https URL</a>}.

BF-SAM: Enhancing SAM Through Multi-Modal Fusion for Fine-Grained Building Function Identification

A multimodal fusion framework for urban scene understanding and functional identification using geospatial data

GeoSAM: Fine-tuning SAM with Multi-Modal Prompts for Mobility Infrastructure Segmentation

Identifying every building's function in large-scale urban areas with multi-modality remote-sensing data

Fine-Grained Building Function Recognition from Street-View Images via Geometry-Aware Semi-Supervised Learning

Zero-Shot Refinement of Buildings' Segmentation Models using SAM

EXPLORING URBAN FUNCTIONAL ZONES BASED ON MULTI-SOURCE SEMANTIC KNOWLEDGE AND CROSS MODAL NETWORK

GSAM: A Deep Neural Network Model for Extracting Computational Representations of Chinese Addresses Fused with Geospatial Feature

A Deep-Learning-Based Multimodal Data Fusion Framework for Urban Region Function Recognition

Building Function Mapping Using Multisource Geospatial Big Data: A Case Study in Shenzhen, China

FusionSAM: Latent Space driven Segment Anything Model for Multimodal Fusion and Segmentation

A Multisource Dynamic Fusion Network for Urban Functional Zone Identification on Remote Sensing, POI, and Building Footprint

MDFF: A Method for Fine-Grained UFZ Mapping with Multimodal Geographic Data and Deep Network

Semantic and Instance Segmentation in Coastal Urban Spatial Perception: A Multi-Task Learning Framework with an Attention Mechanism

Segment Anything with Multiple Modalities

Integrating SAM With Feature Interaction for Remote Sensing Change Detection

Adapting Segment Anything Model to Multi-modal Salient Object Detection with Semantic Feature Fusion Guidance

Simultaneous Extraction of Spatial and Attributional Building Information Across Large-Scale Urban Landscapes from High-Resolution Satellite Imagery

UV-SAM: Adapting Segment Anything Model for Urban Village Identification

Mapping Large-Scale and Fine-Grained Urban Functional Zones from VHR Images Using a Multi-Scale Semantic Segmentation Network and Object Based Approach

BAFNet: Bilateral Attention Fusion Network for Lightweight Semantic Segmentation of Urban Remote Sensing Images