An-Chieh Cheng,Hongxu Yin,Yang Fu,Qiushan Guo,Ruihan Yang,Jan Kautz,Xiaolong Wang,Sifei Liu
Abstract:Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spatial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. SpatialRGPT advances VLMs' spatial understanding through two key innovations: (1) a data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (2) a flexible plugin module for integrating depth information into the visual encoder of existing VLMs. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialRGBT-Bench, a benchmark with ground-truth 3D annotations encompassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in VLMs. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as a region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at <a class="link-external link-https" href="https://www.anjiecheng.me/SpatialRGPT" rel="external noopener nofollow">this https URL</a>
What problem does this paper attempt to address?
The problem that this paper attempts to solve is that the existing Vision Language Models (VLMs) have limited ability in understanding and reasoning about spatial relationships. Although these models perform well in various tasks such as image classification, image captioning, object detection, video understanding, and document parsing, they still have significant difficulties in dealing with simple spatial concepts such as "left", "right", "up", "down" and more complex spatial relationships such as "behind", "in front of", "inside", "outside", "near", "far", etc. These problems not only affect the model's ability to understand the visual environment but also limit its practical applications in fields such as robotics and augmented reality, which require precise spatial awareness to complete tasks such as navigation, operation, and interaction with the real - world environment.
To overcome these challenges, the paper proposes Spatial Region GPT (SpatialRGPT), aiming to enhance the spatial perception and reasoning ability of VLMs through two key innovations:
1. **Data Generation Pipeline**: This pipeline can effectively learn region representations from 3D scene graphs, thereby constructing 3D scene graphs that contain object instances and their spatial relationships.
2. **Flexible Plug - in Module**: This module can integrate depth information into the visual encoders of existing VLMs, improving the model's accurate perception ability of the direction and distance of spatial relationships.
In addition, the paper also proposes a new benchmark test set, SpatialRGBT - Bench, for evaluating the performance of VLMs in 3D spatial cognition. This benchmark test set contains real - world 3D annotations of indoor, outdoor, and simulated environments and can comprehensively evaluate the model's spatial reasoning ability. Through these improvements, SpatialRGPT not only significantly improves its performance in spatial reasoning tasks but also shows strong generalization ability. It can effectively reason about complex spatial relationships without local region prompts and can be applied as a dense reward annotator for region - aware in robotic tasks.