View Selection for 3D Captioning via Diffusion Ranking

Tiange Luo,Justin Johnson,Honglak Lee

2024-04-12

Abstract:Scalable annotation approaches are crucial for constructing extensive 3D-text datasets, facilitating a broader range of applications. However, existing methods sometimes lead to the generation of hallucinated captions, compromising caption quality. This paper explores the issue of hallucination in 3D object captioning, with a focus on Cap3D method, which renders 3D objects into 2D views for captioning using pre-trained models. We pinpoint a major challenge: certain rendered views of 3D objects are atypical, deviating from the training data of standard image captioning models and causing hallucinations. To tackle this, we present DiffuRank, a method that leverages a pre-trained text-to-3D model to assess the alignment between 3D objects and their 2D rendered views, where the view with high alignment closely represent the object's characteristics. By ranking all rendered views and feeding the top-ranked ones into GPT4-Vision, we enhance the accuracy and detail of captions, enabling the correction of 200k captions in the Cap3D dataset and extending it to 1 million captions across Objaverse and Objaverse-XL datasets. Additionally, we showcase the adaptability of DiffuRank by applying it to pre-trained text-to-image models for a Visual Question Answering task, where it outperforms the CLIP model.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper mainly addresses the issues of errors and hallucination in 3D object automatic captioning. The existing method Cap3D renders 3D objects into 2D images and uses a pre-trained model to generate descriptions, but inaccurate descriptions (hallucinations) can occur. Research has found that certain rendered views do not match the training data of the standard image model, leading to the problem. Therefore, the paper proposes the DiffuRank method, which utilizes a pre-trained text-to-3D model to evaluate the alignment between 3D objects and their 2D views, selecting the views that best represent the object features for description. This improves the accuracy and details of the descriptions, correcting about 200,000 descriptions in the Cap3D dataset and expanding it to 1 million descriptions. Additionally, DiffuRank outperforms the CLIP model in visual question answering tasks.