MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors

Yuan Tang,Xu Han,Xianzhi Li,Qiao Yu,Yixue Hao,Long Hu,Min Chen

2024-05-03

Abstract:Large 2D vision-language models (2D-LLMs) have gained significant attention by bridging Large Language Models (LLMs) with images using a simple projector. Inspired by their success, large 3D point cloud-language models (3D-LLMs) also integrate point clouds into LLMs. However, directly aligning point clouds with LLM requires expensive training costs, typically in hundreds of GPU-hours on A100, which hinders the development of 3D-LLMs. In this paper, we introduce MiniGPT-3D, an efficient and powerful 3D-LLM that achieves multiple SOTA results while training for only 27 hours on one RTX 3090. Specifically, we propose to align 3D point clouds with LLMs using 2D priors from 2D-LLMs, which can leverage the similarity between 2D and 3D visual information. We introduce a novel four-stage training strategy for modality alignment in a cascaded way, and a mixture of query experts module to adaptively aggregate features with high efficiency. Moreover, we utilize parameter-efficient fine-tuning methods LoRA and Norm fine-tuning, resulting in only 47.8M learnable parameters, which is up to 260x fewer than existing methods. Extensive experiments show that MiniGPT-3D achieves SOTA on 3D object classification and captioning tasks, with significantly cheaper training costs. Notably, MiniGPT-3D gains an 8.12 increase on GPT-4 evaluation score for the challenging object captioning task compared to ShapeLLM-13B, while the latter costs 160 total GPU-hours on 8 A800. We are the first to explore the efficient 3D-LLM, offering new insights to the community. Code and weights are available at

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

The paper proposes a solution to efficiently align 3D point clouds with Large Language Models (LLMs). Existing 3D Point Cloud Language Models (3D-LLMs) align 3D point clouds directly with LLMs, resulting in high training costs. The paper introduces a new approach called MiniGPT-3D, which utilizes 2D priors (2D prior information from 2D Visual Language Models) to connect 3D point clouds and LLMs more efficiently. MiniGPT-3D gradually transfers the knowledge from 2D-LLMs to 3D space through a four-stage cascaded training strategy and adopts a Mixed Query Expert (MQE) module to efficiently aggregate features in an adaptive manner. Additionally, the model employs parameter-efficient fine-tuning techniques such as LoRA and Norm fine-tuning, reducing the number of learnable parameters. It only takes 27 hours of training on a single RTX 3090 GPU, while existing methods may require hundreds of GPU hours. Experimental results demonstrate that MiniGPT-3D achieves state-of-the-art performance on 3D object classification and caption generation tasks, while significantly reducing training time and resource requirements.