VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation

Xuan He,Dongfu Jiang,Ge Zhang,Max Ku,Achint Soni,Sherman Siu,Haonan Chen,Abhranil Chandra,Ziyan Jiang,Aaran Arulraj,Kai Wang,Quy Duc Do,Yuansheng Ni,Bohan Lyu,Yaswanth Narsupalli,Rongqi Fan,Zhiheng Lyu,Yuchen Lin,Wenhu Chen

2024-10-14

Abstract:The recent years have witnessed great advances in video generation. However, the development of automatic video metrics is lagging significantly behind. None of the existing metric is able to provide reliable scores over generated videos. The main barrier is the lack of large-scale human-annotated dataset. In this paper, we release VideoFeedback, the first large-scale dataset containing human-provided multi-aspect score over 37.6K synthesized videos from 11 existing video generative models. We train VideoScore (initialized from Mantis) based on VideoFeedback to enable automatic video quality assessment. Experiments show that the Spearman correlation between VideoScore and humans can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about 50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and VBench show that VideoScore has consistently much higher correlation with human judges than other metrics. Due to these results, we believe VideoScore can serve as a great proxy for human raters to (1) rate different video models to track progress (2) simulate fine-grained human feedback in Reinforcement Learning with Human Feedback (RLHF) to improve current video generation models.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The main problem this paper attempts to address is the inadequacy of existing automatic evaluation metrics for video generation. Although text-to-video (T2V) generation models have made significant progress in recent years, the automatic metrics used to evaluate the quality of videos generated by these models lag far behind. The existing metrics have the following issues:

1. **Distribution Calculation**: Some metrics require distribution-based calculations and cannot be directly applied to individual model outputs, such as FVD and IS.
2. **Single-Dimension Evaluation**: Most metrics can only evaluate visual quality or text alignment, but cannot cover other important aspects such as motion smoothness, factual consistency, etc.
3. **Lack of Fine-Grained Scoring**: Some metrics only provide an overall score and cannot offer detailed sub-scores in multiple aspects.
4. **Low Correlation with Human Judgment**: Although some studies attempt to evaluate video quality through multimodal large language models (MLLM), these methods have a low correlation with human judgment.

To address these issues, the paper proposes two main contributions:

1. **Constructing a Large-Scale Human-Annotated Dataset**: Created the **VIDEO FEEDBACK** dataset, which includes 37.6K human-annotated synthetic videos from 11 existing T2V generation models.
2. **Training an Automatic Video Evaluation Model**: Based on the **VIDEO FEEDBACK** dataset, trained an automatic video quality evaluation model named **VIDEO SCORE**, which can simulate human feedback and provide fine-grained multi-aspect scoring.

Through these contributions, the paper aims to provide a reliable automatic evaluation tool to help researchers and developers better evaluate and improve T2V generation models.