Segment and Caption Anything

Xiaoke Huang,Jianfeng Wang,Yansong Tang,Zheng Zhang,Han Hu,Jiwen Lu,Lijuan Wang,Zicheng Liu

2024-03-26

Abstract:We propose a method to efficiently equip the Segment Anything Model (SAM) with the ability to generate regional captions. SAM presents strong generalizability to segment anything while is short for semantic understanding. By introducing a lightweight query-based feature mixer, we align the region-specific features with the embedding space of language models for later caption generation. As the number of trainable parameters is small (typically in the order of tens of millions), it costs less computation, less memory usage, and less communication bandwidth, resulting in both fast and scalable training. To address the scarcity problem of regional caption data, we propose to first pre-train our model on objection detection and segmentation tasks. We call this step weak supervision pretraining since the pre-training data only contains category names instead of full-sentence descriptions. The weak supervision pretraining allows us to leverage many publicly available object detection and segmentation datasets. We conduct extensive experiments to demonstrate the superiority of our method and validate each design choice. This work serves as a stepping stone towards scaling up regional captioning data and sheds light on exploring efficient ways to augment SAM with regional semantics. The project page, along with the associated code, can be accessed via <a class="link-external link-https" href="https://xk-huang.github.io/segment-caption-anything/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of how to efficiently endow the Segment Anything Model (SAM) with the ability to generate region descriptions. Specifically, the paper makes the following contributions:

1. **Addressing SAM's lack of semantic understanding**: Although SAM has demonstrated strong generalization capabilities in segmentation tasks, it lacks semantic understanding. By introducing a lightweight query feature mixer, the region-specific features are aligned with the embedding space of the language model, thereby enabling the generation of region descriptions.

2. **Data scarcity issue**: To tackle the scarcity of region description data, the paper proposes a weakly supervised pre-training method. This method first utilizes a large amount of available object detection and segmentation datasets for pre-training, thereby reducing the need for full-sentence descriptions.

3. **Efficient model training**: Due to the relatively small number of newly added trainable parameters (usually in the millions), this method consumes less in terms of computation, memory usage, and communication bandwidth, making the training process both fast and scalable.

4. **Experimental validation**: Extensive experiments validate the effectiveness of this method and demonstrate its state-of-the-art performance on the Visual Genome benchmark (149.8 CIDEr-D, 17.5 METEOR, 31.4 SPICE).

In summary, this paper aims to enhance SAM's regional semantic understanding capabilities through a lightweight approach and explores effective training strategies under large-scale data.