Abstract:Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : <a class="link-external link-https" href="https://vis-www.cs.umass.edu/3dllm/" rel="external noopener nofollow">this https URL</a>.

3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer

LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness

LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

3D-LLM: Injecting the 3D World into Large Language Models

Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding

3D-VLA: A 3D Vision-Language-Action Generative World Model

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

LLMI3D: Empowering LLM with 3D Perception from a Single 2D Image

An Embodied Generalist Agent in 3D World

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks

Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer

LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture

OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving

LLaVA-OneVision: Easy Visual Task Transfer

LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model

LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding