LLaNA: Large Language and NeRF Assistant

Andrea Amaduzzi,Pierluigi Zama Ramirez,Giuseppe Lisanti,Samuele Salti,Luigi Di Stefano

2024-06-18

Abstract:Multimodal Large Language Models (MLLMs) have demonstrated an excellent understanding of images and 3D data. However, both modalities have shortcomings in holistically capturing the appearance and geometry of objects. Meanwhile, Neural Radiance Fields (NeRFs), which encode information within the weights of a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly widespread modality that simultaneously encodes the geometry and photorealistic appearance of objects. This paper investigates the feasibility and effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first general-purpose NeRF-language assistant capable of performing new tasks such as NeRF captioning and Q\&A. Notably, our method directly processes the weights of the NeRF's MLP to extract information about the represented objects without the need to render images or materialize 3D data structures. Moreover, we build a dataset of NeRFs with text annotations for various NeRF-language tasks with no human intervention. Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that processing NeRF weights performs favourably against extracting 2D or 3D representations from NeRFs.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper discusses the possibility of combining Neural Radiance Fields (NeRF) with Large Language Models (LLMs) to create a novel multimodal language model called LLaNA. NeRF is a representation method that can capture the geometric shape and realistic appearance of objects. The researchers found that directly handling the weights of NeRF can effectively extract information about the represented objects without the need for rendering images or constructing 3D data structures.

The main contributions of the paper include:
1. Creating the first general multimodal language model LLaNA that can process NeRF and perform tasks such as NeRF description and question answering.
2. Proposing a method to directly handle the weights of NeRF through a meta-network encoder, which is more efficient than extracting 2D or 3D representations from NeRF.
3. Automatically constructing a NeRF language benchmark dataset and evaluating LLaNA, demonstrating its superiority over existing LLMs that handle discrete representations extracted from NeRF.

In addition, they also discuss the limitations of existing methods, such as the loss of details when rendering NeRF from different perspectives or the lack of detailed 3D geometric shapes extracted. By directly manipulating the NeRF weights, LLaNA can avoid these issues and perform well in understanding and reasoning with NeRF.