GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation

Tong Wu,Guandao Yang,Zhibing Li,Kai Zhang,Ziwei Liu,Leonidas Guibas,Dahua Lin,Gordon Wetzstein

2024-01-10

Abstract:Despite recent advances in text-to-3D generative methods, there is a notable absence of reliable evaluation metrics. Existing metrics usually focus on a single criterion each, such as how well the asset aligned with the input text. These metrics lack the flexibility to generalize to different evaluation criteria and might not align well with human preferences. Conducting user preference studies is an alternative that offers both adaptability and human-aligned results. User studies, however, can be very expensive to scale. This paper presents an automatic, versatile, and human-aligned evaluation metric for text-to-3D generative models. To this end, we first develop a prompt generator using GPT-4V to generate evaluating prompts, which serve as input to compare text-to-3D models. We further design a method instructing GPT-4V to compare two 3D assets according to user-defined criteria. Finally, we use these pairwise comparison results to assign these models Elo ratings. Experimental results suggest our metric strongly align with human preference across different evaluation criteria.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the issue of the lack of reliable evaluation metrics in the field of text-to-3D generation. Existing evaluation metrics typically focus on a single criterion, such as the consistency of the generated 3D assets with the input text, but these metrics lack flexibility, cannot adapt to different evaluation needs, and may not align with human judgment. While user studies can provide adaptability and accuracy, they are costly, difficult to scale, and time-consuming. Therefore, this paper proposes an automatic, flexible, and human-aligned evaluation metric for assessing text-to-3D generation models.

Specifically, the main contributions of the paper include:
1. **Automatic Prompt Generator**: Utilizing GPT-4V to generate input prompts needed for evaluation, which can be customized according to the evaluator's needs, covering different levels of complexity and creativity.
2. **3D Asset Comparator**: Designing a method to guide GPT-4V in comparing two 3D assets based on user-defined criteria.
3. **Elo Rating System**: Using the Elo rating system to assign scores to each model through pairwise comparison results, thereby ranking the models.

The paper experimentally validates the proposed evaluation metrics' high consistency with human judgment across various evaluation criteria, demonstrating their effectiveness and flexibility in evaluating text-to-3D generation models.