ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding

Minghang Zheng,Jiahua Zhang,Qingchao Chen,Yuxin Peng,Yang Liu

DOI: https://doi.org/10.1145/3664647.3681660

2024-08-29

Abstract:Visual grounding aims to localize the object referred to in an image based on a natural language query. Although progress has been made recently, accurately localizing target objects within multiple-instance distractions (multiple objects of the same category as the target) remains a significant challenge. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image, indicating an insufficient understanding of the fine-grained semantics and spatial relationships between objects. In this paper, we propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue. Firstly, we enhance the model's understanding of fine-grained semantics by injecting semantic prior information derived from text queries into the model. This is achieved by leveraging text-to-image generation models to produce images representing the semantic attributes of target objects described in queries. Secondly, we tackle the lack of training samples with multiple distractions by introducing a relation-sensitive data augmentation method. This method generates additional training data by synthesizing images containing multiple objects of the same category and pseudo queries based on their spatial relationships. The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions. We conduct extensive experiments to validate the effectiveness of our methods on five datasets. Code is available at <a class="link-external link-https" href="https://github.com/minghangz/ResVG" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper aims to address the issue of multi-instance interference in visual localization tasks. Specifically:

- **Main Problem**: Current visual localization models perform poorly when there are multiple interfering objects of the same category as the target object in the image. This is because existing models lack fine-grained semantic understanding and fail to comprehend the spatial relationships between objects.

- **Proposed Method**: To solve the above problem, the authors propose a new model named "ResVG". This model improves through two key steps:
1. **Injecting Semantic Prior Information**: Utilizing text-to-image generation models (such as Stable Diffusion) to generate images of the target object based on the query text, thereby enhancing the model's understanding of fine-grained semantics.
2. **Relation-Sensitive Data Augmentation**: Introducing a new data augmentation method that synthesizes images containing multiple objects of the same category and generates pseudo-queries based on the spatial relationships of these objects to improve the model's ability to understand spatial relationships.

Through these two improvements, ResVG significantly enhances the performance of visual localization tasks, especially in scenarios with multiple interfering objects of the same category.