WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Language

Zhenxiang Lin,Xidong Peng,Peishan Cong,Ge Zheng,Yujin Sun,Yuenan Hou,Xinge Zhu,Sibei Yang,Yuexin Ma

2024-07-15

Abstract:We introduce the task of 3D visual grounding in large-scale dynamic scenes based on natural linguistic descriptions and online captured multi-modal visual data, including 2D images and 3D LiDAR point clouds. We present a novel method, dubbed WildRefer, for this task by fully utilizing the rich appearance information in images, the position and geometric clues in point cloud as well as the semantic knowledge of language descriptions. Besides, we propose two novel datasets, i.e., STRefer and LifeRefer, which focus on large-scale human-centric daily-life scenarios accompanied with abundant 3D object and natural language annotations. Our datasets are significant for the research of 3D visual grounding in the wild and has huge potential to boost the development of autonomous driving and service robots. Extensive experiments and ablation studies demonstrate that our method achieves state-of-the-art performance on the proposed benchmarks. The code is provided in <a class="link-external link-https" href="https://github.com/4DVLab/WildRefer" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily aims to address the problem of 3D object localization in large-scale dynamic scenes based on multimodal visual data and natural language descriptions. Specifically, the research proposes a new task—3D Visual Grounding in the Wild (3DVGW), which targets locating specific objects in real-world large-scale dynamic scenes using online-captured multimodal visual data (including 2D images and 3D LiDAR point clouds) and natural language descriptions.

To solve this problem, the paper contributes the following:

1. **Proposes a new method**: Named WildRefer, this method fully leverages appearance information from images, positional and geometric cues from point clouds, and semantic knowledge from language descriptions. This method extracts motion features from continuous visual data by designing a dynamic visual encoder and enhances representation capability through a tri-modal feature interaction module that allows information interaction among language, images, and point clouds.

2. **Constructs two new datasets**: STRefer and LifeRefer, which focus on human-centered daily scenes and contain multimodal visual data and free-form natural language descriptions. These datasets are significant for studying 3D visual localization in wild environments and are expected to advance the development of service robots and autonomous driving technologies.

3. **Achieves state-of-the-art performance**: Extensive experimental validation on the proposed benchmarks shows that the proposed method achieves state-of-the-art performance levels.

In summary, the goal of this paper is to improve the accuracy of object localization in complex dynamic environments by combining multimodal data and natural language processing techniques. This is crucial for developing service robots that can understand natural language instructions and perform tasks in the real world.