Abstract:Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : <a class="link-external link-https" href="https://vis-www.cs.umass.edu/3dllm/" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

This paper proposes a new 3D Language Model (3D-LLMs) aimed at incorporating the 3D world into large-scale language models to address the problem of existing models' limited understanding and processing of rich concepts in the 3D physical world, such as spatial relations, functionality, and physical interaction. By enabling 3D-LLMs to take 3D point clouds and their features as input, the model can perform various 3D-related tasks, such as 3D description, 3D question answering, task decomposition, and more. To overcome the challenges of data acquisition, the researchers designed a unique data generation pipeline that utilizes ChatGPT and three prompt mechanisms to generate the 3D language dataset, covering multiple 3D-related tasks. Additionally, they leveraged 2D Visual Language Models (VLMs) as the foundation and combined it with a 3D feature extractor to effectively train the 3D-LLMs with the support of pre-trained 2D VLMs. To capture 3D spatial information, a 3D localization mechanism was also introduced. The experiments demonstrate that 3D-LLMs outperform baseline models on the ScanQA dataset and perform well on self-built datasets for 3D description, task composition, and 3D-assisted dialogue. The main contributions of the paper include proposing the 3D-LLMs family, innovative data collection methods, 3D feature extraction, and 3D localization mechanism, as well as the planned release of 3D-LLMs, 3D language dataset, and language-aligned 3D features.

3D-LLM: Injecting the 3D World into Large Language Models

3D-LLM: Injecting the 3D World into Large Language Models

When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models

LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding

Language-Image Models with 3D Understanding

LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding

LLMI3D: Empowering LLM with 3D Perception from a Single 2D Image

Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer

Grounded 3D-LLM with Referent Tokens

LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences

3D-VLA: A 3D Vision-Language-Action Generative World Model

LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness

3D Spatial Understanding in MLLMs: Disambiguation and Evaluation

3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination

3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding

SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities

VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification

Unified Scene Representation and Reconstruction for 3D Large Language Models

ShapeLLM: Universal 3D Object Understanding for Embodied Interaction