Abstract:In the realm of human–robot interaction, the integration of visual and verbal cues has become increasingly significant. This paper focuses on the challenges and advancements in referring image segmentation (RIS), a task that involves segmenting images based on textual descriptions. Traditional approaches to RIS have primarily focused on pixel-level classification. These methods, although effective, often overlook the interconnectedness of pixels, which can be crucial for interpreting complex visual scenes. Furthermore, while the PolyFormer model has shown impressive performance in RIS, its large number of parameters and high training data requirements pose significant challenges. These factors restrict its adaptability and optimization on standard consumer hardware, hindering further enhancements in subsequent research. Addressing these issues, our study introduces a novel two-branch decoder framework with SAM (segment anything model) for RIS. This framework incorporates an MLP decoder and a KAN decoder with a multi-scale feature fusion module, enhancing the model's capacity to discern fine details within images. The framework's robustness is further bolstered by an ensemble learning strategy that consolidates the insights from both the MLP and KAN decoder branches. More importantly, we collect the segmentation target edge coordinates and bounding box coordinates as input cues for the SAM model. This strategy leverages SAM's zero-sample learning capabilities to refine and optimize the segmentation outcomes. Our experimental findings, based on the widely recognized RefCOCO, RefCOCO+, and RefCOCOg datasets, confirm the effectiveness of this method. The results not only achieve state-of-the-art (SOTA) performance in segmentation but are also supported by ablation studies that highlight the contributions of each component to the overall improvement in performance.

Beyond Holistic Object Recognition: Enriching Image Understanding with Part States

Semantic Reconstruction based on RGB Image and Sparse Depth

Unsupervised learning of object semantic parts from internal states of CNNs by population encoding

Going Denser with Open-Vocabulary Part Segmentation

Detecting Semantic Parts on Partially Occluded Objects

Towards Multi-Semantic Image Annotation with Graph Regularized Exclusive Group Lasso

Inferring and Leveraging Parts from Object Shape for Improving Semantic Image Synthesis

Unsupervised Part Segmentation through Disentangling Appearance and Shape

OV-PARTS: Towards Open-Vocabulary Part Segmentation

A Hybrid Framework for Referring Image Segmentation: Dual-Decoder Model with SAM Complementation

Learning complementary semantic information for zero-shot recognition

Interpretable and Accurate Fine-grained Recognition via Region Grouping

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts

Exploring Part-Aware Segmentation for Fine-Grained Visual Categorization

Statistical Modeling and Conceptualization of Natural Images

Beyond One-to-One: Rethinking the Referring Image Segmentation

LabelBank: Revisiting Global Perspectives for Semantic Segmentation

Parts4Feature: Learning 3D Global Features from Generally Semantic Parts in Multiple Views

Attribute Recognition from Adaptive Parts

A Synergistical Attention Model for Semantic Segmentation of Remote Sensing Images