AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment

Chunyi Li,Zicheng Zhang,Haoning Wu,Wei Sun,Xiongkuo Min,Xiaohong Liu,Guangtao Zhai,Weisi Lin

DOI: https://doi.org/10.1109/tcsvt.2023.3319020

IF: 5.859

2023-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:With the rapid advancements of the text-to-image generative model, AI-generated images (AGIs) have been widely applied to entertainment, education, social media, etc. However, considering the large quality variance among different AGIs, there is an urgent need for quality models that are consistent with human subjective ratings. To address this issue, we extensively consider various popular AGI models, generated AGI through different prompts and model parameters, and collected subjective scores at the perceptual quality and text-to-image alignment, thus building the most comprehensive AGI subjective quality database AGIQA-3K so far. Furthermore, we conduct a benchmark experiment on this database to evaluate the consistency between the current Image Quality Assessment (IQA) model and human perception, while proposing StairReward that significantly improves the assessment performance of subjective text-to-image alignment. We believe that the fine-grained subjective scores in AGIQA-3K will inspire subsequent AGI quality models to fit human subjective perception mechanisms at both perception and alignment levels and to optimize the generation result of future AGI models. The database is released on https://github.com/lcysyzxdxc/AGIQA-3k-Database.

engineering, electrical & electronic

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address the issue of quality assessment for AI-Generated Images (AGI). Specifically:

1. **Establishing a Comprehensive Quality Database**:
- The paper constructs a database named AGIQA-3K, which includes a large number of AGIs generated from different models (including Generative Adversarial Networks GAN, Autoregressive Models AR, and Diffusion Models). These images have been subjectively rated through standardized experiments.

2. **Refining Quality Dimensions**:
- The database not only considers perceptual quality (such as image clarity, color, etc.) but also pays special attention to text-to-image alignment (i.e., the consistency between the generated image and the input text description). This multi-dimensional evaluation helps to understand the quality of AGI more comprehensively.

3. **Optimizing Evaluation Models**:
- The paper proposes a new evaluation metric called StairReward to improve existing text-to-image alignment evaluation methods. By using refined subjective rating data, the quality of future AGI generation models can be further optimized.

4. **Ensuring Safety**:
- In subjective experiments, participants marked three types of typical unsafe content (social issues, NSFW content, and fake generated images) to ensure the safety of AGI in practical applications.

In summary, this paper is primarily dedicated to constructing a comprehensive and fine-grained AGI quality database and proposing new evaluation methods to promote the development and optimization of AGI generation technology.